nvme/012 & 013: avoid extremely slow xfs IO
PATCH 1/1 nvme/012 & 013: avoid extremely slow xfs IO
It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.
Eric and Dave replied that it is because of too small log size on small disk.
So pass '-l size=32m' to avoid the issue.
With this patch, nvme/012 can be completed in one minute.
--- tests/nvme/012 | 2 +- tests/nvme/013 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
tests/nvme/012 +1 -1
@@ -46,7 +46,7 @@ test() { umount ${mount_dir} > /dev/null 2>&1 - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1 mount /dev/"${nvmedev}n1" "${mount_dir}"
tests/nvme/013 +1 -1
@@ -44,7 +44,7 @@ test() { umount ${mount_dir} > /dev/null 2>&1 - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1 mount /dev/"${nvmedev}n1" "${mount_dir}"
Signature · 1 line
2.9.5
Thanks for the patch Ming. Couple of comments below.
Quoted 10 lines · Ming Lei
It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.
Eric and Dave replied that it is because of too small log size on small disk.
So pass '-l size=32m' to avoid the issue.
With this patch, nvme/012 can be completed in one minute.
Then we should set the QUICK=1 if its taking shorter time.
Quoted 15 lines
Cc Eric Sandeen <[email protected]>Cc Dave Chinner <[email protected]>Cc "Darrick J. Wong" <[email protected]>Signed-off-by Ming Lei <[email protected]>--- tests/nvme/012 | 2 +- tests/nvme/013 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)tests/nvme/012 +0
-0
tests/nvme/012 test() { · line 50
Quoted 6 lines
tests/nvme/012 +1
-1@@ -46,7 +46,7 @@ test() { umount ${mount_dir} > /dev/null 2>&1 - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
As a part of this series lets move this to the helper mkfs in the nvme/rc and use that call in all the file-backed ns related testcases. Let me know if you want me to do that or you would like to do that as a part of this series. I'm okay with anything.
tests/nvme/013 test() { · line 48
Quoted 13 lines
tests/nvme/012 +0
-0mount /dev/"${nvmedev}n1" "${mount_dir}"tests/nvme/013 +1
-1@@ -44,7 +44,7 @@ test() { umount ${mount_dir} > /dev/null 2>&1 - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
Extra space here but can be ignored once we move mkfs to the helper call.
Quoted 3 lines
tests/nvme/013 +0
-0mount /dev/"${nvmedev}n1" "${mount_dir}"
Quoted 58 lines · Chaitanya Kulkarni
Thanks for the patch Ming. Couple of comments below.
Quoted 10 lines · Ming Lei
It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.
Eric and Dave replied that it is because of too small log size on small disk.
So pass '-l size=32m' to avoid the issue.
With this patch, nvme/012 can be completed in one minute.
Then we should set the QUICK=1 if its taking shorter time.
Quoted 15 lines
Cc Eric Sandeen <[email protected]>Cc Dave Chinner <[email protected]>Cc "Darrick J. Wong" <[email protected]>Signed-off-by Ming Lei <[email protected]>--- tests/nvme/012 | 2 +- tests/nvme/013 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)tests/nvme/012 +0
-0tests/nvme/012 test() { · line 50
Quoted 6 lines
tests/nvme/012 +1
-1@@ -46,7 +46,7 @@ test() { umount ${mount_dir} > /dev/null 2>&1 - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1As a part of this series lets move this to the helper mkfs in the nvme/rc and use that call in all the file-backed ns related testcases. Let me know if you want me to do that or you want me to do that as a part of this series. I'm okay with anything.
tests/nvme/013 test() { · line 48
Quoted 13 lines
tests/nvme/012 +0
-0mount /dev/"${nvmedev}n1" "${mount_dir}"tests/nvme/013 +1
-1@@ -44,7 +44,7 @@ test() { umount ${mount_dir} > /dev/null 2>&1 - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1Extra space here but can be ignored once we move mkfs to the helper call.
Quoted 3 lines
tests/nvme/013 +0
-0mount /dev/"${nvmedev}n1" "${mount_dir}"
Quoted 6 lines · Chaitanya Kulkarni
Thanks for the patch Ming. Couple of comments below.
Quoted 3 lines · Ming Lei
It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.
Quoted 8 lines
Quoted 7 lines
Eric and Dave replied that it is because of too small log size on small disk.
So pass '-l size=32m' to avoid the issue.
With this patch, nvme/012 can be completed in one minute.
Then we should set the QUICK=1 if its taking shorter time.
That should have been a QUICK test, it is fine to set it, but it shouldn't belong to this patch.
Quoted 19 lines
Quoted 19 lines
Cc Eric Sandeen <[email protected]>Cc Dave Chinner <[email protected]>Cc "Darrick J. Wong" <[email protected]>Signed-off-by Ming Lei <[email protected]>--- tests/nvme/012 | 2 +- tests/nvme/013 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)tests/nvme/012 +0
-0@@ -46,7 +46,7 @@ test() { umount ${mount_dir} > /dev/null 2>&1
Quoted 7 lines
tests/nvme/012 test() { · line 50
Quoted 3 lines
tests/nvme/012 +1
-1- mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1As a part of this series lets move this to the helper mkfs in the nvme/rc and use that call in all the file-backed ns related testcases. Let me know if you want me to do that or you would like to do that as a part of this series. I'm okay with anything.
I am fine with either way.
Thanks,
Ming Lei
Quoted 47 lines · Ming Lei
Quoted 6 lines · Chaitanya Kulkarni
Thanks for the patch Ming. Couple of comments below.
Quoted 3 lines · Ming Lei
It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.
Quoted 8 lines
Quoted 7 lines
Eric and Dave replied that it is because of too small log size on small disk.
So pass '-l size=32m' to avoid the issue.
With this patch, nvme/012 can be completed in one minute.
Then we should set the QUICK=1 if its taking shorter time.
That should have been a QUICK test, it is fine to set it, but it shouldn't belong to this patch.
Quoted 19 lines
Quoted 19 lines
Cc Eric Sandeen <[email protected]>Cc Dave Chinner <[email protected]>Cc "Darrick J. Wong" <[email protected]>Signed-off-by Ming Lei <[email protected]>--- tests/nvme/012 | 2 +- tests/nvme/013 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)tests/nvme/012 +0
-0@@ -46,7 +46,7 @@ test() { umount ${mount_dir} > /dev/null 2>&1Quoted 7 lines
tests/nvme/012 test() { · line 50
Quoted 3 lines
tests/nvme/012 +1
-1- mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1As a part of this series lets move this to the helper mkfs in the nvme/rc and use that call in all the file-backed ns related testcases. Let me know if you want me to do that or you would like to do that as a part of this series. I'm okay with anything.
Quoted 2 lines
I am fine with either way.
Okay, let's get this in, will send out the refactoring.
Quoted 3 lines
Thanks,
Ming Lei
Quoted 32 lines · Chaitanya Kulkarni
Thanks for the patch Ming. Couple of comments below.
Quoted 10 lines · Ming Lei
It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.
Eric and Dave replied that it is because of too small log size on small disk.
So pass '-l size=32m' to avoid the issue.
With this patch, nvme/012 can be completed in one minute.
Then we should set the QUICK=1 if its taking shorter time.
Quoted 18 lines
Cc Eric Sandeen <[email protected]>Cc Dave Chinner <[email protected]>Cc "Darrick J. Wong" <[email protected]>Signed-off-by Ming Lei <[email protected]>--- tests/nvme/012 | 2 +- tests/nvme/013 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)tests/nvme/012 +0
-0@@ -46,7 +46,7 @@ test() { umount ${mount_dir} > /dev/null 2>&1
Quoted 7 lines
tests/nvme/012 test() { · line 50
Quoted 3 lines
tests/nvme/012 +1
-1- mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1As a part of this series lets move this to the helper mkfs in the nvme/rc and use that call in all the file-backed ns related testcases. Let me know if you want me to do that or you would like to do that as a part of this series. I'm okay with anything.
We also discussed making the log size larger by default, on small fielsystems, but that won't help you yet.
My only caution is that manually setting the log to 32m may actually create a smaller than default log if your backing file happens to be very large. Just something to consider. If the backing files are typically around the 4G size of this test (?) then 32m seems reasonable, 128m certainly would not hurt.
-Eric
Quoted 18 lines
tests/nvme/013 test() { · line 48
Quoted 13 lines
tests/nvme/012 +0
-0mount /dev/"${nvmedev}n1" "${mount_dir}"tests/nvme/013 +1
-1@@ -44,7 +44,7 @@ test() { umount ${mount_dir} > /dev/null 2>&1 - mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1Extra space here but can be ignored once we move mkfs to the helper call.
Quoted 3 lines
tests/nvme/013 +0
-0mount /dev/"${nvmedev}n1" "${mount_dir}"
Quoted 41 lines · Eric Sandeen
Quoted 32 lines · Chaitanya Kulkarni
Thanks for the patch Ming. Couple of comments below.
Quoted 10 lines · Ming Lei
It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.
Eric and Dave replied that it is because of too small log size on small disk.
So pass '-l size=32m' to avoid the issue.
With this patch, nvme/012 can be completed in one minute.
Then we should set the QUICK=1 if its taking shorter time.
Quoted 18 lines
Cc Eric Sandeen <[email protected]>Cc Dave Chinner <[email protected]>Cc "Darrick J. Wong" <[email protected]>Signed-off-by Ming Lei <[email protected]>--- tests/nvme/012 | 2 +- tests/nvme/013 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)tests/nvme/012 +0
-0@@ -46,7 +46,7 @@ test() { umount ${mount_dir} > /dev/null 2>&1Quoted 7 lines
tests/nvme/012 test() { · line 50
Quoted 3 lines
tests/nvme/012 +1
-1- mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1 + mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1As a part of this series lets move this to the helper mkfs in the nvme/rc and use that call in all the file-backed ns related testcases. Let me know if you want me to do that or you would like to do that as a part of this series. I'm okay with anything.
Quoted 8 lines
We also discussed making the log size larger by default, on small fielsystems, but that won't help you yet.
My only caution is that manually setting the log to 32m may actually create a smaller than default log if your backing file happens to be very large. Just something to consider. If the backing files are typically around the 4G size of this test (?) then 32m seems reasonable, 128m certainly would not hurt.
In the two tests, the device size is 1G, and looks 32m log size works just fine.
thanks,
Ming
Quoted 9 lines · Ming Lei
It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.
Eric and Dave replied that it is because of too small log size on small disk.
So pass '-l size=32m' to avoid the issue.
With this patch, nvme/012 can be completed in one minute.
Nice:
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed]
runtime 402.605s ... 27.431s
Thanks, Ming, applied.