Skip to contentSkip to conversation

generic/551: improve error reporting and handling

PATCH In review Naohiro Aota 2 participants 7 messages Aug 22, 2023 – Aug 29, 2023

lore
Share & tools
lore
#0 lore raw

cover generic/551: improve error reporting and handling

When generic/551 failed, it is difficult to investigate the root cause because (1) it continues working after an error, corrupting the disk state,
(2) there is no detail info printed, and (3) aio-dio-write-verify failed to check the event errors.

So, this series addresses these issues.

Naohiro Aota (3):
aio-dio-write-verify: check for the IO errors
aio-dio-write-verify: print more info on the error case generic/551: bail out test if aio-dio-write-verify failed

 src/aio-dio-regress/aio-dio-write-verify.c | 17 ++++++++++++++---
 tests/generic/551                          |  6 ++++--
 2 files changed, 18 insertions(+), 5 deletions(-)
Signature · 1 line
2.41.0
#1 lore raw

PATCH 1/3 aio-dio-write-verify: check for the IO errors

The async write IOs can return some errors, which may lead to a short read or corruption in io_verify() stage. Catch an error early to identify the root cause easily.

---
 src/aio-dio-regress/aio-dio-write-verify.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
src/aio-dio-regress/aio-dio-write-verify.c +10 -0
index 302b8fe4a096..90e41b391289 100644
@@ -238,6 +238,16 @@ static int io_write(int fd, int num_events)
 		return 1;
 	}
 
+	for (i = 0; i < num_events; i++) {
+		int err = (int)evs[i].res;
+
+		if (err < 0) {
+			fprintf(stderr, "error %s with event %d\n",
+				strerror(err), i);
+			return 1;
+		}
+	}
+
 	/* Try to destroy at here, not necessary, so don't check result */
 	io_destroy(ctx);
 
Signature · 1 line
2.41.0
Reviewed-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] aio-dio-write-verify: check for the IO errors

Quoted 6 lines · Naohiro Aota

The async write IOs can return some errors, which may lead to a short read or corruption in io_verify() stage. Catch an error early to identify the root cause easily.

---

Looks good to me,

Quoted 27 lines
 src/aio-dio-regress/aio-dio-write-verify.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
src/aio-dio-regress/aio-dio-write-verify.c +10 -0
index 302b8fe4a096..90e41b391289 100644
@@ -238,6 +238,16 @@ static int io_write(int fd, int num_events)
 		return 1;
 	}
 
+	for (i = 0; i < num_events; i++) {
+		int err = (int)evs[i].res;
+
+		if (err < 0) {
+			fprintf(stderr, "error %s with event %d\n",
+				strerror(err), i);
+			return 1;
+		}
+	}
+
 	/* Try to destroy at here, not necessary, so don't check result */
 	io_destroy(ctx);
 
Signature · 1 line
2.41.0
#3 lore raw

PATCH 2/3 aio-dio-write-verify: print more info on the error case

In reply to Naohiro Aota

When short read or corruption happened, it is difficult to locate which IO event failed. Print the address to make it identifiable.

---
 src/aio-dio-regress/aio-dio-write-verify.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
src/aio-dio-regress/aio-dio-write-verify.c +4 -3
index 90e41b391289..dabbfacde3f8 100644
@@ -267,13 +267,14 @@ static int io_verify(int fd)
 			perror("pread");
 			return 1;
 		} else if (sret != p->param->buf_size) {
-			fprintf(stderr, "short read %zd was less than %zu\n",
-			        sret, p->param->buf_size);
+			fprintf(stderr, "short read %zd was less than %zu at %zu\n",
+			        sret, p->param->buf_size, p->param->offset);
 			return 1;
 		}
 		if (memcmp(p->param->buf,
 		           p->param->cmp_buf, p->param->buf_size)) {
-			printf("Find corruption\n");
+			printf("Find corruption at %zu length %zu\n", p->param->offset,
+			       p->param->buf_size);
 			dump_buffer(p->param->buf, p->param->offset,
 			            p->param->buf_size);
 			corrupted++;
Signature · 1 line
2.41.0
Reviewed-by on v1 2/3
#4 lore raw

Re: [PATCH 2/3] aio-dio-write-verify: print more info on the error case

Quoted 5 lines · Naohiro Aota

When short read or corruption happened, it is difficult to locate which IO event failed. Print the address to make it identifiable.

---

Good to me,

Quoted 28 lines
 src/aio-dio-regress/aio-dio-write-verify.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
src/aio-dio-regress/aio-dio-write-verify.c +4 -3
index 90e41b391289..dabbfacde3f8 100644
@@ -267,13 +267,14 @@ static int io_verify(int fd)
 			perror("pread");
 			return 1;
 		} else if (sret != p->param->buf_size) {
-			fprintf(stderr, "short read %zd was less than %zu\n",
-			        sret, p->param->buf_size);
+			fprintf(stderr, "short read %zd was less than %zu at %zu\n",
+			        sret, p->param->buf_size, p->param->offset);
 			return 1;
 		}
 		if (memcmp(p->param->buf,
 		           p->param->cmp_buf, p->param->buf_size)) {
-			printf("Find corruption\n");
+			printf("Find corruption at %zu length %zu\n", p->param->offset,
+			       p->param->buf_size);
 			dump_buffer(p->param->buf, p->param->offset,
 			            p->param->buf_size);
 			corrupted++;
Signature · 1 line
2.41.0
#5 lore raw

PATCH 3/3 generic/551: bail out test if aio-dio-write-verify failed

In reply to Naohiro Aota

When the AIO program failed, it is better to bail out the test to keep the failed state intact.

---
 tests/generic/551 | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tests/generic/551 +4 -2
index 2689270aa3dd..58c7f9a516c7 100755
@@ -56,16 +56,18 @@ do_test()
 	truncsize=$(((RANDOM * diosize + RANDOM % diosize) % max_io_size_b))
 
 	$AIO_TEST -t $truncsize $oper_list $localfile
-	if [ $? -ne 0 ];then
+	ret=$?
+	if [ $ret -ne 0 ];then
 		echo "$AIO_TEST -t $truncsize $oper_list $localfile"
 		echo "==========^^ Fail ^^=========="
 	fi
+	return $ret
 }
 
 testimes=$((LOAD_FACTOR * 100))
 while [ $testimes -gt 0 ]; do
 	echo > $localfile
-	do_test
+	do_test || break
 	((testimes--))
 done
 
Signature · 1 line
2.41.0
Reviewed-by on v1 3/3
#6 lore raw

Re: [PATCH 3/3] generic/551: bail out test if aio-dio-write-verify failed

Quoted 5 lines · Naohiro Aota

When the AIO program failed, it is better to bail out the test to keep the failed state intact.

---

Good to me,

Quoted 31 lines
 tests/generic/551 | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tests/generic/551 +4 -2
index 2689270aa3dd..58c7f9a516c7 100755
@@ -56,16 +56,18 @@ do_test()
 	truncsize=$(((RANDOM * diosize + RANDOM % diosize) % max_io_size_b))
 
 	$AIO_TEST -t $truncsize $oper_list $localfile
-	if [ $? -ne 0 ];then
+	ret=$?
+	if [ $ret -ne 0 ];then
 		echo "$AIO_TEST -t $truncsize $oper_list $localfile"
 		echo "==========^^ Fail ^^=========="
 	fi
+	return $ret
 }
 
 testimes=$((LOAD_FACTOR * 100))
 while [ $testimes -gt 0 ]; do
 	echo > $localfile
-	do_test
+	do_test || break
 	((testimes--))
 done
 
Signature · 1 line
2.41.0