Skip to contentSkip to conversation

Non sequential linked chains and IO_LINK support

DISCUSSION Bhatia, Sumeet 2 participants 4 messages May 6, 2020 – May 7, 2020 io_uring

lore
Share & tools
lore
#0 lore raw

Hello everyone,

I've been exploring iouring to submit disk operations. My application generates disk operations based on some events and operations are unknown until those events occur. Some of these disk operations are interdependent others are not.

Example: Following operations are generated and submitted before any of them are complete operation_0 (independent operation)
operation_1 (independent operation),​
operation_2 (to be issued only if operation_0 was successful),
operation_3 (independent operation),
operation_4 (to be issued only if operation_1 was successful)

In my example I have two independent link chains, (operation_0, operation_2) and (operation_1, operation_4). iouring documentation suggests IOSQE_IO_LINK expects link chains to be sequential and will not support my use case.

I explored creating new iouring context for each of these linked chains. But it turns out depending on disk size there can be somewhere between 500-1000 such chains. I'm not sure whether it is prudent to create that many iouring contexts.

I am reaching out to check whether there would be a generic need to support nonsequential linked chains on a single iouring context. Would love to hear all your thoughts.

Thanks,
Sumeet

#1 lore raw
Quoted 4 lines · Sumeet

Hello everyone,

I've been exploring iouring to submit disk operations. My application generates disk operations based on some events and operations are unknown until those events occur. Some of these disk operations are interdependent others are not.

Quoted 8 lines

Example: Following operations are generated and submitted before any of them are complete operation_0 (independent operation)
operation_1 (independent operation),​
operation_2 (to be issued only if operation_0 was successful),
operation_3 (independent operation),
operation_4 (to be issued only if operation_1 was successful)

In my example I have two independent link chains, (operation_0, operation_2) and (operation_1, operation_4). iouring documentation suggests IOSQE_IO_LINK expects link chains to be sequential and will not support my use case.

First of all, there shouldn't be a submission (i.e. io_uring_enter(to_submit>0)) between adding linked requests to a submission queue (SQ). It'd be racy otherwise.

E.g. you can't do:

add_sqe(op0)
submit(op0)
add_sqe(op2, linked)

Though the following is valid, as we don't submit op0:

add_sqe(opX)
add_sqe(op0)
submit(up until opX)
add_sqe(op2, linked)

And that means you can reorder them just before submitting, or filing them into the SQ in a better order.

Is it helpful? Let's figure out how to cover your case.

Quoted 1 line

I explored creating new iouring context for each of these linked chains. But it turns out depending on disk size there can be somewhere between 500-1000 such chains. I'm not sure whether it is prudent to create that many iouring contexts.

Then you would need to wait on them (e.g. epoll or 1000 threads), and that would defeat the whole idea. In any case even with sharing io-wq and having small CQ and SQ, it'd be wasteful keeping many resources duplicated.

Quoted 6 lines

I am reaching out to check whether there would be a generic need to support nonsequential linked chains on a single iouring context. Would love to hear all your thoughts.

Thanks,
Sumeet

Signature · 1 line
Pavel Begunkov
#2 lore raw

Thank you for the response!

Unfortunately when the application submits operation_0 it has no way of determining if/when operation_2 would be generated.

For now I plan to maintain a list of outstanding operations. If operation_2 gets generated while operation_0 is in flight the application will hold its submission until operation_0 is completed.

I wanted to check whether this would be a generic use case and would warrant native support in iouring?

Thanks,
Sumeet
________________________________________
From: Pavel Begunkov <[email protected]>
Sent: Wednesday, May 6, 2020 6:11 PM
To: Bhatia, Sumeet; [email protected]
Cc: Hegde, Pramod; Jens Axboe
Subject: RE: [EXTERNAL] Non sequential linked chains and IO_LINK support

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 4 lines · Sumeet

Hello everyone,

I've been exploring iouring to submit disk operations. My application generates disk operations based on some events and operations are unknown until those events occur. Some of these disk operations are interdependent others are not.

Quoted 8 lines

Example: Following operations are generated and submitted before any of them are complete operation_0 (independent operation)
operation_1 (independent operation),​
operation_2 (to be issued only if operation_0 was successful),
operation_3 (independent operation),
operation_4 (to be issued only if operation_1 was successful)

In my example I have two independent link chains, (operation_0, operation_2) and (operation_1, operation_4). iouring documentation suggests IOSQE_IO_LINK expects link chains to be sequential and will not support my use case.

First of all, there shouldn't be a submission (i.e. io_uring_enter(to_submit>0)) between adding linked requests to a submission queue (SQ). It'd be racy otherwise.

E.g. you can't do:

add_sqe(op0)
submit(op0)
add_sqe(op2, linked)

Though the following is valid, as we don't submit op0:

add_sqe(opX)
add_sqe(op0)
submit(up until opX)
add_sqe(op2, linked)

And that means you can reorder them just before submitting, or filing them into the SQ in a better order.

Is it helpful? Let's figure out how to cover your case.

Quoted 1 line

I explored creating new iouring context for each of these linked chains. But it turns out depending on disk size there can be somewhere between 500-1000 such chains. I'm not sure whether it is prudent to create that many iouring contexts.

Then you would need to wait on them (e.g. epoll or 1000 threads), and that would defeat the whole idea. In any case even with sharing io-wq and having small CQ and SQ, it'd be wasteful keeping many resources duplicated.

Quoted 6 lines

I am reaching out to check whether there would be a generic need to support nonsequential linked chains on a single iouring context. Would love to hear all your thoughts.

Thanks,
Sumeet

--
Pavel Begunkov

#3 lore raw
Quoted 5 lines · Sumeet

Thank you for the response!

Unfortunately when the application submits operation_0 it has no way of determining if/when operation_2 would be generated.

For now I plan to maintain a list of outstanding operations. If operation_2 gets generated while operation_0 is in flight the application will hold its submission until operation_0 is completed.

Yes, that's the general workflow. Links in current form are not very helpful for your case. Those who don't care about latency are trying to throttle requests a bit, to collect several of them and send at once.

Quoted 1 line

I wanted to check whether this would be a generic use case and would warrant native support in iouring?

Trying to link to a request that was already submitted, would just complicate the kernel and userspace as well, I don't think it's viable to support that, unless there will be huge benefit.

On the other hand, this is an interesting case for our BPF ideas. E.g. you load a BPF program, which on request completion will poll your internal queue for a specific disk and submit requests from there. Kind of custom polling. We don't have BPF in io_uring yet, though.

Quoted 66 lines

Thanks,
Sumeet
________________________________________
From: Pavel Begunkov <[email protected]>
Sent: Wednesday, May 6, 2020 6:11 PM
To: Bhatia, Sumeet; [email protected]
Cc: Hegde, Pramod; Jens Axboe
Subject: RE: [EXTERNAL] Non sequential linked chains and IO_LINK support

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 4 lines · Sumeet

Hello everyone,

I've been exploring iouring to submit disk operations. My application generates disk operations based on some events and operations are unknown until those events occur. Some of these disk operations are interdependent others are not.

Quoted 8 lines

Example: Following operations are generated and submitted before any of them are complete operation_0 (independent operation)
operation_1 (independent operation),​
operation_2 (to be issued only if operation_0 was successful),
operation_3 (independent operation),
operation_4 (to be issued only if operation_1 was successful)

In my example I have two independent link chains, (operation_0, operation_2) and (operation_1, operation_4). iouring documentation suggests IOSQE_IO_LINK expects link chains to be sequential and will not support my use case.

First of all, there shouldn't be a submission (i.e. io_uring_enter(to_submit>0)) between adding linked requests to a submission queue (SQ). It'd be racy otherwise.

E.g. you can't do:

add_sqe(op0)
submit(op0)
add_sqe(op2, linked)

Though the following is valid, as we don't submit op0:

add_sqe(opX)
add_sqe(op0)
submit(up until opX)
add_sqe(op2, linked)

And that means you can reorder them just before submitting, or filing them into the SQ in a better order.

Is it helpful? Let's figure out how to cover your case.

Quoted 1 line

I explored creating new iouring context for each of these linked chains. But it turns out depending on disk size there can be somewhere between 500-1000 such chains. I'm not sure whether it is prudent to create that many iouring contexts.

Then you would need to wait on them (e.g. epoll or 1000 threads), and that would defeat the whole idea. In any case even with sharing io-wq and having small CQ and SQ, it'd be wasteful keeping many resources duplicated.

Quoted 6 lines

I am reaching out to check whether there would be a generic need to support nonsequential linked chains on a single iouring context. Would love to hear all your thoughts.

Thanks,
Sumeet

--
Pavel Begunkov

Signature · 1 line
Pavel Begunkov