Skip to contentSkip to conversation

Fix NBD TLS iotests on RHEL-7

PATCH Superseded Daniel P. Berrangé 3 participants 8 messages Feb 19, 2019

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Fix NBD TLS iotests on RHEL-7

This fixes a failure of iotest 233 due to certtool problesm in RHEL7 wrt to SIGPIPE

Daniel P. Berrangé (2):
iotests: ensure we print nbd server log on error
iotests: avoid broken pipe with certtool

 tests/qemu-iotests/233        |  3 +++
 tests/qemu-iotests/common.tls | 48 +++++++++++++++++++++++------------
 2 files changed, 35 insertions(+), 16 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/2 iotests: ensure we print nbd server log on error

If we abort the iotest early the server.log file might contain useful information for diagnosing the problem. Ensure its contents are displayed in this case.

---
 tests/qemu-iotests/233 | 3 +++
 1 file changed, 3 insertions(+)
tests/qemu-iotests/233 +3 -0
index fc345a1a46..27932df075 100755
@@ -30,6 +30,8 @@ _cleanup()
 {
     nbd_server_stop
     _cleanup_test_img
+    # If we aborted early we want to see this log for diagnosis
+    test -f "$TEST_DIR/server.log" && cat "$TEST_DIR/server.log"
     rm -f "$TEST_DIR/server.log"
     tls_x509_cleanup
 }
@@ -120,6 +122,7 @@ $QEMU_IO -f $IMGFMT -r -U -c 'r -P 0x22 1m 1m' "$TEST_IMG" | _filter_qemu_io
 echo
 echo "== final server log =="
 cat "$TEST_DIR/server.log"
+rm -f $TEST_DIR/server.log
 
 # success, all done
 echo "*** done"
Signature · 1 line
2.20.1
Reviewed-by on v1 1/2
#2 lore raw

Re: [Qemu-devel] [PATCH 1/2] iotests: ensure we print nbd server log on error

Quoted 8 lines · Daniel P. Berrangé

If we abort the iotest early the server.log file might contain useful information for diagnosing the problem. Ensure its contents are displayed in this case.

---
 tests/qemu-iotests/233 | 3 +++
 1 file changed, 3 insertions(+)
Quoted 23 lines
tests/qemu-iotests/233 +3 -0
index fc345a1a46..27932df075 100755
@@ -30,6 +30,8 @@ _cleanup()
 {
     nbd_server_stop
     _cleanup_test_img
+    # If we aborted early we want to see this log for diagnosis
+    test -f "$TEST_DIR/server.log" && cat "$TEST_DIR/server.log"
     rm -f "$TEST_DIR/server.log"
     tls_x509_cleanup
 }
@@ -120,6 +122,7 @@ $QEMU_IO -f $IMGFMT -r -U -c 'r -P 0x22 1m 1m' "$TEST_IMG" | _filter_qemu_io
 echo
 echo "== final server log =="
 cat "$TEST_DIR/server.log"
+rm -f $TEST_DIR/server.log
 
 # success, all done
 echo "*** done"
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#3 lore raw

PATCH 2/2 iotests: avoid broken pipe with certtool

In reply to Daniel P. Berrangé

When we run "certtool | head -1" the latter command is likely to complete and exit before certtool has written everything it wants to stderr. In at least the RHEL-7 gnutls 3.3.29 this causes certtool to quit with broken pipe before it has finished writing the desired output file to disk. This causes non-deterministic failures of the iotest 233 because the certs are sometimes zero length files. If certtool fails the "head -1" means we also loose any useful error message it would have printed.

Thus this patch gets rid of the pipe and post-processes the output in a more flexible & reliable manner.

---
 tests/qemu-iotests/common.tls | 48 +++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 16 deletions(-)
tests/qemu-iotests/common.tls +32 -16
index eae81789bb..6cf11ed383 100644
@@ -29,6 +29,17 @@ tls_x509_cleanup()
 }
 
 
+tls_certtool()
+{
+    certtool "$@" 1>certtool.log 2>&1
+    if test "$?" = 0; then
+      head -1 certtool.log
+    else
+      cat certtool.log
+    fi
+    rm -f certtool.log
+}
+
 tls_x509_init()
 {
     (certtool --help) >/dev/null 2>&1 || \
@@ -71,10 +82,11 @@ ca
 cert_signing_key
 EOF
 
-    certtool --generate-self-signed \
-             --load-privkey "${tls_dir}/key.pem" \
-             --template "${tls_dir}/ca.info" \
-             --outfile "${tls_dir}/$name-cert.pem" 2>&1 | head -1
+    tls_certtool \
+        --generate-self-signed \
+        --load-privkey "${tls_dir}/key.pem" \
+        --template "${tls_dir}/ca.info" \
+        --outfile "${tls_dir}/$name-cert.pem"
 
     rm -f "${tls_dir}/ca.info"
 }
@@ -98,12 +110,14 @@ encryption_key
 signing_key
 EOF
 
-    certtool --generate-certificate \
-             --load-ca-privkey "${tls_dir}/key.pem" \
-             --load-ca-certificate "${tls_dir}/$caname-cert.pem" \
-             --load-privkey "${tls_dir}/key.pem" \
-             --template "${tls_dir}/cert.info" \
-             --outfile "${tls_dir}/$name/server-cert.pem" 2>&1 | head -1
+    tls_certtool \
+        --generate-certificate \
+        --load-ca-privkey "${tls_dir}/key.pem" \
+        --load-ca-certificate "${tls_dir}/$caname-cert.pem" \
+        --load-privkey "${tls_dir}/key.pem" \
+        --template "${tls_dir}/cert.info" \
+        --outfile "${tls_dir}/$name/server-cert.pem"
+
     ln -s "${tls_dir}/$caname-cert.pem" "${tls_dir}/$name/ca-cert.pem"
     ln -s "${tls_dir}/key.pem" "${tls_dir}/$name/server-key.pem"
 
@@ -127,12 +141,14 @@ encryption_key
 signing_key
 EOF
 
-    certtool --generate-certificate \
-             --load-ca-privkey "${tls_dir}/key.pem" \
-             --load-ca-certificate "${tls_dir}/$caname-cert.pem" \
-             --load-privkey "${tls_dir}/key.pem" \
-             --template "${tls_dir}/cert.info" \
-             --outfile "${tls_dir}/$name/client-cert.pem" 2>&1 | head -1
+    tls_certtool \
+        --generate-certificate \
+        --load-ca-privkey "${tls_dir}/key.pem" \
+        --load-ca-certificate "${tls_dir}/$caname-cert.pem" \
+        --load-privkey "${tls_dir}/key.pem" \
+        --template "${tls_dir}/cert.info" \
+        --outfile "${tls_dir}/$name/client-cert.pem"
+
     ln -s "${tls_dir}/$caname-cert.pem" "${tls_dir}/$name/ca-cert.pem"
     ln -s "${tls_dir}/key.pem" "${tls_dir}/$name/client-key.pem"
 
Signature · 1 line
2.20.1
Tested-by on v1 2/2
#4 lore raw

Re: [Qemu-devel] [PATCH 2/2] iotests: avoid broken pipe with certtool

Quoted 26 lines · Daniel P. Berrangé

When we run "certtool | head -1" the latter command is likely to complete and exit before certtool has written everything it wants to stderr. In at least the RHEL-7 gnutls 3.3.29 this causes certtool to quit with broken pipe before it has finished writing the desired output file to disk. This causes non-deterministic failures of the iotest 233 because the certs are sometimes zero length files. If certtool fails the "head -1" means we also loose any useful error message it would have printed.

Thus this patch gets rid of the pipe and post-processes the output in a more flexible & reliable manner.

---
 tests/qemu-iotests/common.tls | 48 +++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 16 deletions(-)
tests/qemu-iotests/common.tls +0 -0
index eae81789bb..6cf11ed383 100644
@@ -29,6 +29,17 @@ tls_x509_cleanup()
 }
 
 

tests/qemu-iotests/common.tls tls_x509_cleanup() · line 41

Quoted 10 lines
tests/qemu-iotests/common.tls +10 -0
+tls_certtool()
+{
+    certtool "$@" 1>certtool.log 2>&1
+    if test "$?" = 0; then
+      head -1 certtool.log
+    else
+      cat certtool.log
+    fi
+    rm -f certtool.log
+}

I assume this is running in a unique directory so that there can not be a clash with a test running in parallel? Otherwise I'd recommend an mktemp file name here...

Anyway, this fixes the issue for me, too, thus:

#5 lore raw

Re: [Qemu-devel] [PATCH 2/2] iotests: avoid broken pipe with certtool

Quoted 27 lines · Thomas Huth
Quoted 26 lines · Daniel P. Berrangé

When we run "certtool | head -1" the latter command is likely to complete and exit before certtool has written everything it wants to stderr. In at least the RHEL-7 gnutls 3.3.29 this causes certtool to quit with broken pipe before it has finished writing the desired output file to disk. This causes non-deterministic failures of the iotest 233 because the certs are sometimes zero length files. If certtool fails the "head -1" means we also loose any useful error message it would have printed.

Thus this patch gets rid of the pipe and post-processes the output in a more flexible & reliable manner.

---
 tests/qemu-iotests/common.tls | 48 +++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 16 deletions(-)
tests/qemu-iotests/common.tls +0 -0
index eae81789bb..6cf11ed383 100644
@@ -29,6 +29,17 @@ tls_x509_cleanup()
 }
 
 
Quoted 14 lines

tests/qemu-iotests/common.tls tls_x509_cleanup() · line 41

Quoted 10 lines
tests/qemu-iotests/common.tls +10 -0
+tls_certtool()
+{
+    certtool "$@" 1>certtool.log 2>&1
+    if test "$?" = 0; then
+      head -1 certtool.log
+    else
+      cat certtool.log
+    fi
+    rm -f certtool.log
+}

I assume this is running in a unique directory so that there can not be a clash with a test running in parallel? Otherwise I'd recommend an mktemp file name here...

Hmm, could be a problem. Whatever happened to the plan to make every iotests run in a private directory ? Did that ever get done ?

Quoted 4 lines

Anyway, this fixes the issue for me, too, thus:

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#6 lore raw

Re: [Qemu-devel] [PATCH 2/2] iotests: avoid broken pipe with certtool

Quoted 15 lines · Daniel P. Berrangé
Quoted 6 lines
Quoted 6 lines
tests/qemu-iotests/common.tls +6 -0
+tls_certtool()
+{
+    certtool "$@" 1>certtool.log 2>&1
+    if test "$?" = 0; then
+      head -1 certtool.log
+    else
Quoted 8 lines

tests/qemu-iotests/common.tls tls_x509_cleanup() · line 41

Quoted 4 lines
tests/qemu-iotests/common.tls +4 -0
+      cat certtool.log
+    fi
+    rm -f certtool.log
+}

I assume this is running in a unique directory so that there can not be a clash with a test running in parallel? Otherwise I'd recommend an mktemp file name here...

Quoted 2 lines

Hmm, could be a problem. Whatever happened to the plan to make every iotests run in a private directory ? Did that ever get done ?

Nope, still not done :(

Jeff left Red Hat, and so far no one else has revived his series. Volunteers?

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
Reviewed-by on v1 2/2
#7 lore raw

Re: [Qemu-devel] [PATCH 2/2] iotests: avoid broken pipe with certtool

In reply to Daniel P. Berrangé
Quoted 9 lines · Daniel P. Berrangé

When we run "certtool | head -1" the latter command is likely to complete and exit before certtool has written everything it wants to stderr. In at least the RHEL-7 gnutls 3.3.29 this causes certtool to quit with broken pipe before it has finished writing the desired output file to disk. This causes non-deterministic failures of the iotest 233 because the certs are sometimes zero length files. If certtool fails the "head -1" means we also loose any useful error message it would have printed.

Quoted 3 lines

Thus this patch gets rid of the pipe and post-processes the output in a more flexible & reliable manner.

Better than my attempt.

Quoted 5 lines
---
 tests/qemu-iotests/common.tls | 48 +++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 16 deletions(-)

As Thomas pointed out, it is not parallel-safe, but that's a bigger issue with all of the iotests, so not this patch's problem.

tests/qemu-iotests/common.tls tls_x509_cleanup() · line 34

Quoted 12 lines
tests/qemu-iotests/common.tls +3 -0
index eae81789bb..6cf11ed383 100644
@@ -29,6 +29,17 @@ tls_x509_cleanup()
 }
 
 
+tls_certtool()
+{
+    certtool "$@" 1>certtool.log 2>&1

I tend to use '>' instead of '1>', but they are identical.

tests/qemu-iotests/common.tls tls_x509_cleanup() · line 36

Quoted 2 lines
tests/qemu-iotests/common.tls +2 -0
+    if test "$?" = 0; then
+      head -1 certtool.log

Technically, POSIX says 'head -1' is not portable (it was historical practice, but a wording change in POSIX 2001 made it invalid, which wasn't fixed until POSIX 2008 - and there are historical versions of GNU coreutils which went out of their way to reject the usage, although that has since been fixed); the portable spelling these days is 'head -n1'. But as the test was already using 'head -1', I'm not going to make you switch it.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org