Skip to contentSkip to conversation

nvme/012 & 013: avoid extremely slow xfs IO

PATCH In review Ming Lei 5 participants 8 messages Apr 15, 2019 – Apr 17, 2019 Block layer

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

PATCH 1/1 nvme/012 & 013: avoid extremely slow xfs IO

It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.

Eric and Dave replied that it is because of too small log size on small disk.

So pass '-l size=32m' to avoid the issue.

With this patch, nvme/012 can be completed in one minute.

Cc Eric Sandeen <[email protected]>
Cc Dave Chinner <[email protected]>
Cc "Darrick J. Wong" <[email protected]>
---
 tests/nvme/012 | 2 +-
 tests/nvme/013 | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
tests/nvme/012 +1 -1
index 9a6801511df7..d7a8751ec752 100755
@@ -46,7 +46,7 @@ test() {
 
 	umount ${mount_dir} > /dev/null 2>&1
 
-	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
+	mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
 
 	mount /dev/"${nvmedev}n1" "${mount_dir}"
 
tests/nvme/013 +1 -1
index e346cd4baa35..bc435f752dca 100755
@@ -44,7 +44,7 @@ test() {
 
 	umount ${mount_dir} > /dev/null 2>&1
 
-	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
+	mkfs.xfs -l size=32m  -f /dev/"${nvmedev}n1" > /dev/null 2>&1
 
 	mount /dev/"${nvmedev}n1" "${mount_dir}"
 
Signature · 1 line
2.9.5
#1 lore raw

Thanks for the patch Ming. Couple of comments below.

Quoted 10 lines · Ming Lei

It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.

Eric and Dave replied that it is because of too small log size on small disk.

So pass '-l size=32m' to avoid the issue.

With this patch, nvme/012 can be completed in one minute.

Then we should set the QUICK=1 if its taking shorter time.

Quoted 15 lines
Cc Eric Sandeen <[email protected]>
Cc Dave Chinner <[email protected]>
Cc "Darrick J. Wong" <[email protected]>
---
  tests/nvme/012 | 2 +-
  tests/nvme/013 | 2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)
tests/nvme/012 +0 -0
index 9a6801511df7..d7a8751ec752 100755

tests/nvme/012 test() { · line 50

Quoted 6 lines
tests/nvme/012 +1 -1
@@ -46,7 +46,7 @@ test() {
 
  	umount ${mount_dir} > /dev/null 2>&1
 
-	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
+	mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
 

As a part of this series lets move this to the helper mkfs in the nvme/rc and use that call in all the file-backed ns related testcases. Let me know if you want me to do that or you would like to do that as a part of this series. I'm okay with anything.

tests/nvme/013 test() { · line 48

Quoted 13 lines
tests/nvme/012 +0 -0
  	mount /dev/"${nvmedev}n1" "${mount_dir}"
 
tests/nvme/013 +1 -1
index e346cd4baa35..bc435f752dca 100755
@@ -44,7 +44,7 @@ test() {
 
  	umount ${mount_dir} > /dev/null 2>&1
 
-	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
+	mkfs.xfs -l size=32m  -f /dev/"${nvmedev}n1" > /dev/null 2>&1
 

Extra space here but can be ignored once we move mkfs to the helper call.

Quoted 3 lines
tests/nvme/013 +0 -0
  	mount /dev/"${nvmedev}n1" "${mount_dir}"
 
#2 lore raw
Quoted 58 lines · Chaitanya Kulkarni

Thanks for the patch Ming. Couple of comments below.

Quoted 10 lines · Ming Lei

It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.

Eric and Dave replied that it is because of too small log size on small disk.

So pass '-l size=32m' to avoid the issue.

With this patch, nvme/012 can be completed in one minute.

Then we should set the QUICK=1 if its taking shorter time.

Quoted 15 lines
Cc Eric Sandeen <[email protected]>
Cc Dave Chinner <[email protected]>
Cc "Darrick J. Wong" <[email protected]>
---
   tests/nvme/012 | 2 +-
   tests/nvme/013 | 2 +-
   2 files changed, 2 insertions(+), 2 deletions(-)
tests/nvme/012 +0 -0
index 9a6801511df7..d7a8751ec752 100755

tests/nvme/012 test() { · line 50

Quoted 6 lines
tests/nvme/012 +1 -1
@@ -46,7 +46,7 @@ test() {
 
   	umount ${mount_dir} > /dev/null 2>&1
 
-	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
+	mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
 

As a part of this series lets move this to the helper mkfs in the nvme/rc and use that call in all the file-backed ns related testcases. Let me know if you want me to do that or you want me to do that as a part of this series. I'm okay with anything.

tests/nvme/013 test() { · line 48

Quoted 13 lines
tests/nvme/012 +0 -0
   	mount /dev/"${nvmedev}n1" "${mount_dir}"
 
tests/nvme/013 +1 -1
index e346cd4baa35..bc435f752dca 100755
@@ -44,7 +44,7 @@ test() {
 
   	umount ${mount_dir} > /dev/null 2>&1
 
-	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
+	mkfs.xfs -l size=32m  -f /dev/"${nvmedev}n1" > /dev/null 2>&1
 

Extra space here but can be ignored once we move mkfs to the helper call.

Quoted 3 lines
tests/nvme/013 +0 -0
   	mount /dev/"${nvmedev}n1" "${mount_dir}"
 
#3 lore raw
In reply to Chaitanya Kulkarni
Quoted 6 lines · Chaitanya Kulkarni

Thanks for the patch Ming. Couple of comments below.

Quoted 3 lines · Ming Lei

It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.

Quoted 8 lines
Quoted 7 lines

Eric and Dave replied that it is because of too small log size on small disk.

So pass '-l size=32m' to avoid the issue.

With this patch, nvme/012 can be completed in one minute.

Then we should set the QUICK=1 if its taking shorter time.

That should have been a QUICK test, it is fine to set it, but it shouldn't belong to this patch.

Quoted 19 lines
Quoted 19 lines
Cc Eric Sandeen <[email protected]>
Cc Dave Chinner <[email protected]>
Cc "Darrick J. Wong" <[email protected]>
---
  tests/nvme/012 | 2 +-
  tests/nvme/013 | 2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)
tests/nvme/012 +0 -0
index 9a6801511df7..d7a8751ec752 100755
@@ -46,7 +46,7 @@ test() {
 
      umount ${mount_dir} > /dev/null 2>&1
 
Quoted 7 lines

tests/nvme/012 test() { · line 50

Quoted 3 lines
tests/nvme/012 +1 -1
-     mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
+     mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
 

As a part of this series lets move this to the helper mkfs in the nvme/rc and use that call in all the file-backed ns related testcases. Let me know if you want me to do that or you would like to do that as a part of this series. I'm okay with anything.

I am fine with either way.

Thanks,
Ming Lei

#4 lore raw
Quoted 47 lines · Ming Lei
Quoted 6 lines · Chaitanya Kulkarni

Thanks for the patch Ming. Couple of comments below.

Quoted 3 lines · Ming Lei

It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.

Quoted 8 lines
Quoted 7 lines

Eric and Dave replied that it is because of too small log size on small disk.

So pass '-l size=32m' to avoid the issue.

With this patch, nvme/012 can be completed in one minute.

Then we should set the QUICK=1 if its taking shorter time.

That should have been a QUICK test, it is fine to set it, but it shouldn't belong to this patch.

Quoted 19 lines
Quoted 19 lines
Cc Eric Sandeen <[email protected]>
Cc Dave Chinner <[email protected]>
Cc "Darrick J. Wong" <[email protected]>
---
   tests/nvme/012 | 2 +-
   tests/nvme/013 | 2 +-
   2 files changed, 2 insertions(+), 2 deletions(-)
tests/nvme/012 +0 -0
index 9a6801511df7..d7a8751ec752 100755
@@ -46,7 +46,7 @@ test() {
 
       umount ${mount_dir} > /dev/null 2>&1
 
Quoted 7 lines

tests/nvme/012 test() { · line 50

Quoted 3 lines
tests/nvme/012 +1 -1
-     mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
+     mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
 

As a part of this series lets move this to the helper mkfs in the nvme/rc and use that call in all the file-backed ns related testcases. Let me know if you want me to do that or you would like to do that as a part of this series. I'm okay with anything.

Quoted 2 lines

I am fine with either way.

Okay, let's get this in, will send out the refactoring.

Quoted 3 lines

Thanks,
Ming Lei

#5 lore raw
In reply to Chaitanya Kulkarni
Quoted 32 lines · Chaitanya Kulkarni

Thanks for the patch Ming. Couple of comments below.

Quoted 10 lines · Ming Lei

It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.

Eric and Dave replied that it is because of too small log size on small disk.

So pass '-l size=32m' to avoid the issue.

With this patch, nvme/012 can be completed in one minute.

Then we should set the QUICK=1 if its taking shorter time.

Quoted 18 lines
Cc Eric Sandeen <[email protected]>
Cc Dave Chinner <[email protected]>
Cc "Darrick J. Wong" <[email protected]>
---
  tests/nvme/012 | 2 +-
  tests/nvme/013 | 2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)
tests/nvme/012 +0 -0
index 9a6801511df7..d7a8751ec752 100755
@@ -46,7 +46,7 @@ test() {
 
  	umount ${mount_dir} > /dev/null 2>&1
 
Quoted 7 lines

tests/nvme/012 test() { · line 50

Quoted 3 lines
tests/nvme/012 +1 -1
-	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
+	mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
 

As a part of this series lets move this to the helper mkfs in the nvme/rc and use that call in all the file-backed ns related testcases. Let me know if you want me to do that or you would like to do that as a part of this series. I'm okay with anything.

We also discussed making the log size larger by default, on small fielsystems, but that won't help you yet.

My only caution is that manually setting the log to 32m may actually create a smaller than default log if your backing file happens to be very large. Just something to consider. If the backing files are typically around the 4G size of this test (?) then 32m seems reasonable, 128m certainly would not hurt.

-Eric

Quoted 18 lines

tests/nvme/013 test() { · line 48

Quoted 13 lines
tests/nvme/012 +0 -0
  	mount /dev/"${nvmedev}n1" "${mount_dir}"
 
tests/nvme/013 +1 -1
index e346cd4baa35..bc435f752dca 100755
@@ -44,7 +44,7 @@ test() {
 
  	umount ${mount_dir} > /dev/null 2>&1
 
-	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
+	mkfs.xfs -l size=32m  -f /dev/"${nvmedev}n1" > /dev/null 2>&1
 

Extra space here but can be ignored once we move mkfs to the helper call.

Quoted 3 lines
tests/nvme/013 +0 -0
  	mount /dev/"${nvmedev}n1" "${mount_dir}"
 
author
#6 lore raw
Quoted 41 lines · Eric Sandeen
Quoted 32 lines · Chaitanya Kulkarni

Thanks for the patch Ming. Couple of comments below.

Quoted 10 lines · Ming Lei

It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.

Eric and Dave replied that it is because of too small log size on small disk.

So pass '-l size=32m' to avoid the issue.

With this patch, nvme/012 can be completed in one minute.

Then we should set the QUICK=1 if its taking shorter time.

Quoted 18 lines
Cc Eric Sandeen <[email protected]>
Cc Dave Chinner <[email protected]>
Cc "Darrick J. Wong" <[email protected]>
---
  tests/nvme/012 | 2 +-
  tests/nvme/013 | 2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)
tests/nvme/012 +0 -0
index 9a6801511df7..d7a8751ec752 100755
@@ -46,7 +46,7 @@ test() {
 
  	umount ${mount_dir} > /dev/null 2>&1
 
Quoted 7 lines

tests/nvme/012 test() { · line 50

Quoted 3 lines
tests/nvme/012 +1 -1
-	mkfs.xfs -f /dev/"${nvmedev}n1" > /dev/null 2>&1
+	mkfs.xfs -l size=32m -f /dev/"${nvmedev}n1" > /dev/null 2>&1
 

As a part of this series lets move this to the helper mkfs in the nvme/rc and use that call in all the file-backed ns related testcases. Let me know if you want me to do that or you would like to do that as a part of this series. I'm okay with anything.

Quoted 8 lines

We also discussed making the log size larger by default, on small fielsystems, but that won't help you yet.

My only caution is that manually setting the log to 32m may actually create a smaller than default log if your backing file happens to be very large. Just something to consider. If the backing files are typically around the 4G size of this test (?) then 32m seems reasonable, 128m certainly would not hurt.

In the two tests, the device size is 1G, and looks 32m log size works just fine.

thanks,
Ming

#7 lore raw
In reply to Ming Lei
Quoted 9 lines · Ming Lei

It is observed that nvme/012 may take ~17 minutes to complete on aarch64, even worse it may trigger IO timeout on nvme-loop.

Eric and Dave replied that it is because of too small log size on small disk.

So pass '-l size=32m' to avoid the issue.

With this patch, nvme/012 can be completed in one minute.

Nice:

nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed]

    runtime  402.605s  ...  27.431s

Thanks, Ming, applied.