Skip to contentSkip to conversation

fixed files

RFC Pavel Begunkov 2 participants 5 messages Feb 8, 2020 – Feb 9, 2020 io_uring

lore
Share & tools
lore
#0 lore raw

Hi,

As you remember, splice(2) needs two fds, and it's a bit of a pain finding a place for the second REQ_F_FIXED_FILE flag. So, I was thinking, can we use the last (i.e. sign) bit to mark an fd as fixed? A lot of userspace programs consider any negative result of open() as an error, so it's more or less safe to reuse it.

e.g.
fill_sqe(fd) // is not fixed
fill_sqe(buf_idx | LAST_BIT) // fixed file

Signature · 1 line
Pavel Begunkov
maintainer
#1 lore raw
Quoted 11 lines · Pavel Begunkov

Hi,

As you remember, splice(2) needs two fds, and it's a bit of a pain finding a place for the second REQ_F_FIXED_FILE flag. So, I was thinking, can we use the last (i.e. sign) bit to mark an fd as fixed? A lot of userspace programs consider any negative result of open() as an error, so it's more or less safe to reuse it.

e.g.
fill_sqe(fd) // is not fixed
fill_sqe(buf_idx | LAST_BIT) // fixed file

Right now we only support 1024 fixed buffers anyway, so we do have some space there. If we steal a bit, it'll still allow us to expand to 32K of fixed buffers in the future.

It's a bit iffy, but like you, I don't immediately see a better way to do this that doesn't include stealing an IOSQE bit or adding a special splice flag for it. Might still prefer the latter, to be honest...

Signature · 1 line
Jens Axboe
#2 lore raw
Quoted 13 lines · Jens Axboe
Quoted 11 lines · Pavel Begunkov

Hi,

As you remember, splice(2) needs two fds, and it's a bit of a pain finding a place for the second REQ_F_FIXED_FILE flag. So, I was thinking, can we use the last (i.e. sign) bit to mark an fd as fixed? A lot of userspace programs consider any negative result of open() as an error, so it's more or less safe to reuse it.

e.g.
fill_sqe(fd) // is not fixed
fill_sqe(buf_idx | LAST_BIT) // fixed file

Quoted 7 lines

Right now we only support 1024 fixed buffers anyway, so we do have some space there. If we steal a bit, it'll still allow us to expand to 32K of fixed buffers in the future.

It's a bit iffy, but like you, I don't immediately see a better way to do this that doesn't include stealing an IOSQE bit or adding a special splice flag for it. Might still prefer the latter, to be honest...

"fixed" is clearly a per-{fd,buffer} attribute. If I'd now design it from the scratch, I would store fixed-resource index in the same field as fds and addr (but not separate @buf_index), and have per-resource switch-flag somewhere. And then I see 2 convenient ways:

1. encode the fixed bit into addr and fd, as supposed above.

2. Add N generic IOSQE_FIXED bits (i.e. IOSQE_FIXED_RESOURSE{1,2,...}), which correspond to resources (fd, buffer, etc) in order of occurrence in an sqe. I wouldn't expect having more than 3-4 flags.

And then IORING_OP_{READ,WRITE}_FIXED would have been the same opcode as the corresponding non-fixed version. But backward-compatibility is a pain.

Signature · 1 line
Pavel Begunkov
maintainer
#3 lore raw
Quoted 27 lines · Pavel Begunkov
Quoted 13 lines · Jens Axboe
Quoted 11 lines · Pavel Begunkov

Hi,

As you remember, splice(2) needs two fds, and it's a bit of a pain finding a place for the second REQ_F_FIXED_FILE flag. So, I was thinking, can we use the last (i.e. sign) bit to mark an fd as fixed? A lot of userspace programs consider any negative result of open() as an error, so it's more or less safe to reuse it.

e.g.
fill_sqe(fd) // is not fixed
fill_sqe(buf_idx | LAST_BIT) // fixed file

Quoted 7 lines

Right now we only support 1024 fixed buffers anyway, so we do have some space there. If we steal a bit, it'll still allow us to expand to 32K of fixed buffers in the future.

It's a bit iffy, but like you, I don't immediately see a better way to do this that doesn't include stealing an IOSQE bit or adding a special splice flag for it. Might still prefer the latter, to be honest...

"fixed" is clearly a per-{fd,buffer} attribute. If I'd now design it from the scratch, I would store fixed-resource index in the same field as fds and addr (but not separate @buf_index), and have per-resource switch-flag somewhere. And then I see 2 convenient ways:

Quoted 8 lines

1. encode the fixed bit into addr and fd, as supposed above.

2. Add N generic IOSQE_FIXED bits (i.e. IOSQE_FIXED_RESOURSE{1,2,...}), which correspond to resources (fd, buffer, etc) in order of occurrence in an sqe. I wouldn't expect having more than 3-4 flags.

And then IORING_OP_{READ,WRITE}_FIXED would have been the same opcode as the corresponding non-fixed version. But backward-compatibility is a pain.

It's always much easier looking back, hindsight is much clearer. I'd also expand the sqe flags bits to 16 at least, but oh well.

I do think that for this particular case we add a SPLICE_F_FD1_FIXED and ditto for fd2, and just have the direct splice/vmsplice syscalls reject them as invalid. Both splice and vmsplice -EINVAL for unknown flags, which makes this possible.

That seems cleaner to me than trying to shoe-horn this information into the sqe itself, and it can easily be done as a prep patch to adding splice support.

Signature · 1 line
Jens Axboe
#4 lore raw
Quoted 34 lines · Jens Axboe
Quoted 33 lines · Pavel Begunkov
Quoted 13 lines · Jens Axboe
Quoted 11 lines · Pavel Begunkov

Hi,

As you remember, splice(2) needs two fds, and it's a bit of a pain finding a place for the second REQ_F_FIXED_FILE flag. So, I was thinking, can we use the last (i.e. sign) bit to mark an fd as fixed? A lot of userspace programs consider any negative result of open() as an error, so it's more or less safe to reuse it.

e.g.
fill_sqe(fd) // is not fixed
fill_sqe(buf_idx | LAST_BIT) // fixed file

Quoted 7 lines

Right now we only support 1024 fixed buffers anyway, so we do have some space there. If we steal a bit, it'll still allow us to expand to 32K of fixed buffers in the future.

It's a bit iffy, but like you, I don't immediately see a better way to do this that doesn't include stealing an IOSQE bit or adding a special splice flag for it. Might still prefer the latter, to be honest...

"fixed" is clearly a per-{fd,buffer} attribute. If I'd now design it from the scratch, I would store fixed-resource index in the same field as fds and addr (but not separate @buf_index), and have per-resource switch-flag somewhere. And then I see 2 convenient ways:

1. encode the fixed bit into addr and fd, as supposed above.

2. Add N generic IOSQE_FIXED bits (i.e. IOSQE_FIXED_RESOURSE{1,2,...}), which correspond to resources (fd, buffer, etc) in order of occurrence in an sqe. I wouldn't expect having more than 3-4 flags.

Quoted 5 lines
Quoted 2 lines

And then IORING_OP_{READ,WRITE}_FIXED would have been the same opcode as the corresponding non-fixed version. But backward-compatibility is a pain.

It's always much easier looking back, hindsight is much clearer. I'd also expand the sqe flags bits to 16 at least, but oh well.

Totally agree. And I didn't meant I could have done better

BTW, we can extend sqe flags into free bits at the end of sqe, would need a bit of packing/unpacking though.
e.g. int flags = sqe1_flags | (sqe2_flags << 8)

Quoted 4 lines

I do think that for this particular case we add a SPLICE_F_FD1_FIXED and ditto for fd2, and just have the direct splice/vmsplice syscalls reject them as invalid. Both splice and vmsplice -EINVAL for unknown flags, which makes this possible.

Yes, I remember that from the splice thread.

Quoted 4 lines

That seems cleaner to me than trying to shoe-horn this information into the sqe itself, and it can easily be done as a prep patch to adding splice support.

Signature · 1 line
Pavel Begunkov