Skip to contentSkip to conversation

Use direct I/O for loop devices

PATCH In review Demi Marie Obenour 3 participants 6 messages Aug 17, 2022 – Oct 20, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Use direct I/O for loop devices

This is a huge performance improvement for two reasons:

1. It uses the filesystem’s asynchronous I/O support, rather than using synchronous I/O.
2. It bypasses the page cache, removing a redundant layer of caching and associated overhead.

---
 tools/hotplug/Linux/block | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/hotplug/Linux/block +1 -1
index 2691b56951c9b82094471a141b9e0bed04abb929..75785f3a5422c4a5f962a4cd4f6acae5080d036d 100644
@@ -330,7 +330,7 @@ mount it read-write in a guest domain."
         else
           roflag=''
         fi
-        do_or_die losetup $roflag "$loopdev" "$file"
+        do_or_die losetup --direct-io=on $roflag "$loopdev" "$file"
         xenstore_write "$XENBUS_PATH/node" "$loopdev"
         write_dev "$loopdev"
         release_lock "block"
Signature · 3 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab
#1 lore raw
Quoted 15 lines · Demi Marie Obenour

This is a huge performance improvement for two reasons:

1. It uses the filesystem’s asynchronous I/O support, rather than using synchronous I/O.
2. It bypasses the page cache, removing a redundant layer of caching and associated overhead.

---
 tools/hotplug/Linux/block | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/hotplug/Linux/block +0 -0
index 2691b56951c9b82094471a141b9e0bed04abb929..75785f3a5422c4a5f962a4cd4f6acae5080d036d 100644

tools/hotplug/Linux/block mount it read-write in a guest domain." · line 333

Quoted 5 lines
tools/hotplug/Linux/block +1 -1
@@ -330,7 +330,7 @@ mount it read-write in a guest domain."
         else
           roflag=''
         fi
-        do_or_die losetup $roflag "$loopdev" "$file"
+        do_or_die losetup --direct-io=on $roflag "$loopdev" "$file"

I guess you want to first check (maybe in tools/configure) that losetup actually supports that option. The old-ish one I'm looking at doesn't, according to its --help output at least.

Jan

#2 lore raw
Quoted 11 lines · Jan Beulich
Quoted 10 lines · Demi Marie Obenour

This is a huge performance improvement for two reasons:

1. It uses the filesystem’s asynchronous I/O support, rather than using synchronous I/O.
2. It bypasses the page cache, removing a redundant layer of caching and associated overhead.

---
 tools/hotplug/Linux/block | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 14 lines

tools/hotplug/Linux/block mount it read-write in a guest domain." · line 333

Quoted 10 lines
tools/hotplug/Linux/block +1 -1
index 2691b56951c9b82094471a141b9e0bed04abb929..75785f3a5422c4a5f962a4cd4f6acae5080d036d 100644
@@ -330,7 +330,7 @@ mount it read-write in a guest domain."
         else
           roflag=''
         fi
-        do_or_die losetup $roflag "$loopdev" "$file"
+        do_or_die losetup --direct-io=on $roflag "$loopdev" "$file"

I guess you want to first check (maybe in tools/configure) that losetup actually supports that option. The old-ish one I'm looking at doesn't, according to its --help output at least.

What version are you referring to?

In Qubes OS the current plan is to use a block “script” written in C, to improve performance and (when combined with kernel patches) eliminate race conditions. This code could be made a wrapper for the C version.

Signature · 3 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab
#3 lore raw
Quoted 15 lines · Demi Marie Obenour
Quoted 13 lines · Jan Beulich

tools/hotplug/Linux/block mount it read-write in a guest domain." · line 333

Quoted 8 lines · Demi Marie Obenour
tools/hotplug/Linux/block +1 -1
@@ -330,7 +330,7 @@ mount it read-write in a guest domain."
         else
           roflag=''
         fi
-        do_or_die losetup $roflag "$loopdev" "$file"
+        do_or_die losetup --direct-io=on $roflag "$loopdev" "$file"

I guess you want to first check (maybe in tools/configure) that losetup actually supports that option. The old-ish one I'm looking at doesn't, according to its --help output at least.

Quoted 1 line

What version are you referring to?

The tool itself doesn't recognize --version. It originates from util-linux 2.19.1 from all I can tell.

Quoted 3 lines

In Qubes OS the current plan is to use a block “script” written in C, to improve performance and (when combined with kernel patches) eliminate race conditions. This code could be made a wrapper for the C version.

Is this relevant here in some way I don't recognize, or did you say this only to provide some further background info?

Jan

#4 lore raw
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Demi Marie Obenour
Quoted 13 lines · Jan Beulich

tools/hotplug/Linux/block mount it read-write in a guest domain." · line 333

Quoted 8 lines · Demi Marie Obenour
tools/hotplug/Linux/block +1 -1
@@ -330,7 +330,7 @@ mount it read-write in a guest domain."
         else
           roflag=''
         fi
-        do_or_die losetup $roflag "$loopdev" "$file"
+        do_or_die losetup --direct-io=on $roflag "$loopdev" "$file"

I guess you want to first check (maybe in tools/configure) that losetup actually supports that option. The old-ish one I'm looking at doesn't, according to its --help output at least.

Quoted 4 lines
Quoted 1 line

What version are you referring to?

The tool itself doesn't recognize --version. It originates from util-linux 2.19.1 from all I can tell.

I see. Qubes OS doesn’t need to support such old versions. Are you referring to

Quoted 6 lines
Quoted 3 lines

In Qubes OS the current plan is to use a block “script” written in C, to improve performance and (when combined with kernel patches) eliminate race conditions. This code could be made a wrapper for the C version.

Is this relevant here in some way I don't recognize, or did you say this only to provide some further background info?

If the C code becomes part of the toolstack it would avoid needing to call into losetup.

Signature · 3 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab
#5 lore raw
In reply to Demi Marie Obenour
Quoted 6 lines · Demi Marie Obenour

This is a huge performance improvement for two reasons:

1. It uses the filesystem’s asynchronous I/O support, rather than using synchronous I/O.
2. It bypasses the page cache, removing a redundant layer of caching and associated overhead.

Cross-posting here just in case: this patch breaks block script on devices with 4k block size, at least with some util-linux versions. More details at https://github.com/QubesOS/qubes-issues/issues/7828

Quoted 22 lines
---
 tools/hotplug/Linux/block | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/hotplug/Linux/block +1 -1
index 2691b56951c9b82094471a141b9e0bed04abb929..75785f3a5422c4a5f962a4cd4f6acae5080d036d 100644
@@ -330,7 +330,7 @@ mount it read-write in a guest domain."
         else
           roflag=''
         fi
-        do_or_die losetup $roflag "$loopdev" "$file"
+        do_or_die losetup --direct-io=on $roflag "$loopdev" "$file"
         xenstore_write "$XENBUS_PATH/node" "$loopdev"
         write_dev "$loopdev"
         release_lock "block"
Signature · 3 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab
Signature · 3 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab