CQE repeats the first item?
I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description
I'm using syscalls to call io_uring_setup and io_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again.
In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with different user_data (0x1234 + i), and I used the opcode IORING_OP_NOP.
I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has the user_data as '18446744073709551615' which is correct, but the first 10 all has user_data be 0x1234 which is weird AF since only one item has that user_data and I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter once
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615
Jens Axboe
maintainerQuoted 15 lines · Jacob Thompson
I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call
io_uring_setupandio_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with differentuser_data(0x1234+ i), and I used the opcodeIORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has theuser_dataas '18446744073709551615' which is correct, but the first 10 all hasuser_databe0x1234which is weird AF since only one item has thatuser_dataand I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter once
Quoted 11 lines
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615
You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.
Signature · 1 line
Jens Axboe
Quoted 28 lines · Jens Axboe
Quoted 15 lines · Jacob Thompson
I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call
io_uring_setupandio_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with differentuser_data(0x1234+ i), and I used the opcodeIORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has theuser_dataas '18446744073709551615' which is correct, but the first 10 all hasuser_databe0x1234which is weird AF since only one item has thatuser_dataand I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter onceQuoted 11 lines
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615
Quoted 8 lines
You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call
io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.Signature · 1 line
Jens Axboe
I tried reproducing it in a smaller file. Assume I did everything wrong but somehow I seem to get results and they're not correct.
The codebase I'd like to use this in has very little activity (could go seconds without a single syscall), then execute a few hundreds-thousand (which I like to be async). SQPOLL sounds like the one best for my usecase. You can see I updated the sq tail before enter and I used IORING_ENTER_SQ_WAKEUP + slept for a second. The sq tail isn't zero which means I have results? and you can see its 10 of the same user_data
cq head is 0 enter result was 10
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
FFFFFFFF -1
#include <unistd.h>
#include <stdio.h>
#include <assert.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/syscall.h>
#include <linux/io_uring.h>
int io_uring_setup(unsigned entries, io_uring_params*params) { return syscall(__NR_io_uring_setup, entries, params); }
int io_uring_enter(int ring_fd, unsigned int to_submit, unsigned int min_complete, unsigned int flags, void*sig) { return (int) syscall(__NR_io_uring_enter, ring_fd, to_submit, min_complete, flags, 0); }
typedef int* IntPtr;
#define X(NAME) NAME = (int*)(p+params.sq_off.NAME) struct sqinfo
{
IntPtr head, tail, ring_mask, ring_entries, flags, dropped;
int*array;
void set(char*p, io_uring_params¶ms) { X(head); X(tail); X(ring_mask); X(ring_entries); X(flags); X(dropped); array = (int*)(p+params.sq_off.array); }
};
#undef X
#define X(NAME) NAME = (int*)(p+params.cq_off.NAME)
struct cqinfo
{
IntPtr head, tail, ring_mask, ring_entries, overflow, flags;
io_uring_cqe*cqes;
void set(char*p, io_uring_params¶ms) { X(head); X(tail); X(ring_mask); X(ring_entries); X(overflow); X(flags); cqes = (io_uring_cqe*)(p+params.cq_off.cqes); }
};#undef X
int main(int argc, char*argv[])
{
int queue_size = 256;
io_uring_params param{}; // zero init
param.flags = IORING_SETUP_SQPOLL;
int ringFD = io_uring_setup(queue_size, ¶m);
assert(ringFD>0);
assert(param.features & IORING_FEAT_SINGLE_MMAP);
auto base_length = param.sq_off.array + param.sq_entries*4;
char *base = (char*) mmap(0, base_length, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, ringFD, IORING_OFF_SQ_RING);
assert(base != MAP_FAILED);
auto sqes = (io_uring_sqe*)mmap(0, param.sq_entries * sizeof(io_uring_sqe), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, ringFD, IORING_OFF_SQES);
assert(sqes != MAP_FAILED);
cqinfo cq;
sqinfo sq;
cq.set(base, param);
sq.set(base, param);
for(int i=0; i<*sq.ring_entries; i++) {
memset(sqes+i, -1, sizeof(io_uring_sqe));
}
for(int i=0; i<*cq.ring_entries; i++) {
memset(cq.cqes+i, -1, sizeof(io_uring_cqe));
}
// Take 10 items
assert(*sq.tail == 0);
auto p = &sqes[0];
memset(p, 0, sizeof(io_uring_sqe)*10);
for(int i=0; i<10; i++) {
sqes[i].opcode = IORING_OP_NOP;
sqes[i].user_data = 0x1234+i;
}
__atomic_store_n(sq.tail, 10, __ATOMIC_RELEASE);
//int res = io_uring_enter(ringFD, 10, 10, IORING_ENTER_SQ_WAIT, 0);
int res = io_uring_enter(ringFD, 10, 10, IORING_ENTER_SQ_WAKEUP, 0);
sleep(1);
printf("cq tail is %d enter result was %d\n", __atomic_load_n(cq.tail, __ATOMIC_ACQUIRE), res);
for(int i=0; i<11; i++) {
printf("%X %d\n", cq.cqes[i].user_data, cq.cqes[i].res);
}
return 0;
}Jens Axboe
maintainerQuoted 44 lines · Jacob Thompson
Quoted 28 lines · Jens Axboe
Quoted 15 lines · Jacob Thompson
I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call
io_uring_setupandio_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with differentuser_data(0x1234+ i), and I used the opcodeIORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has theuser_dataas '18446744073709551615' which is correct, but the first 10 all hasuser_databe0x1234which is weird AF since only one item has thatuser_dataand I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter onceQuoted 11 lines
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615Quoted 8 lines
You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call
io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.Signature · 1 line
Jens AxboeI tried reproducing it in a smaller file. Assume I did everything wrong but somehow I seem to get results and they're not correct.
The codebase I'd like to use this in has very little activity (could go seconds without a single syscall), then execute a few hundreds-thousand (which I like to be async). SQPOLL sounds like the one best for my usecase. You can see I updated the sq tail before enter and I used
IORING_ENTER_SQ_WAKEUP+ slept for a second. The sq tail isn't zero which means I have results? and you can see its 10 of the sameuser_data
Quoted 12 lines
cq head is 0 enter result was 10
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
FFFFFFFF -1
I looked at your test code, and you're setting up 10 NOP requests with userdata == 0x1234, and hence you get 10 completions with that userdata. For some reason you iterate 11 CQEs, which means your last one is the one that you already filled with -1.
In other words, it very much looks like it's working as it should. Any reason why you're using the raw interface rather than liburing? All of this seems to be not understanding how the ring works, and liburing helps isolate you from that. The SQ ring doesn't tell you anything about whether you have results (CQEs?), the difference between the SQ head and tail just tell you if there's something to submit. The CQ ring head and tail would tell you if there are CQEs to reap or not.
Signature · 1 line
Jens Axboe
Quoted 63 lines · Jens Axboe
Quoted 44 lines · Jacob Thompson
Quoted 28 lines · Jens Axboe
Quoted 15 lines · Jacob Thompson
I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call
io_uring_setupandio_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with differentuser_data(0x1234+ i), and I used the opcodeIORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has theuser_dataas '18446744073709551615' which is correct, but the first 10 all hasuser_databe0x1234which is weird AF since only one item has thatuser_dataand I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter onceQuoted 11 lines
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615Quoted 8 lines
You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call
io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.Signature · 1 line
Jens AxboeI tried reproducing it in a smaller file. Assume I did everything wrong but somehow I seem to get results and they're not correct.
The codebase I'd like to use this in has very little activity (could go seconds without a single syscall), then execute a few hundreds-thousand (which I like to be async). SQPOLL sounds like the one best for my usecase. You can see I updated the sq tail before enter and I used
IORING_ENTER_SQ_WAKEUP+ slept for a second. The sq tail isn't zero which means I have results? and you can see its 10 of the sameuser_dataQuoted 12 lines
cq head is 0 enter result was 10
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
FFFFFFFF -1I looked at your test code, and you're setting up 10 NOP requests with userdata ==
0x1234, and hence you get 10 completions with that userdata. For some reason you iterate 11 CQEs, which means your last one is the one that you already filled with -1.
Quoted 10 lines
In other words, it very much looks like it's working as it should. Any reason why you're using the raw interface rather than liburing? All of this seems to be not understanding how the ring works, and liburing helps isolate you from that. The SQ ring doesn't tell you anything about whether you have results (CQEs?), the difference between the SQ head and tail just tell you if there's something to submit. The CQ ring head and tail would tell you if there are CQEs to reap or not.
Signature · 1 line
Jens Axboe
You must be seeing something that I'm not. I had a +i in the line, should the user_data not increment every item? The line was 'sqes[i].user_data = 0x1234+i;'. The 11th iteration is intentional to see the value of the memset earlier.
Quoted 1 line
I'd encourage you to read and understand liburing, even if you aren't going to use it. That will tell you how to reap and iterate CQE events.
I did. I didn't try to compile and step through but I'm in the middle of rereading the manual. I want to finish it before I look at the lib again
Jens Axboe
maintainerQuoted 72 lines · Jacob Thompson
Quoted 71 lines · Jens Axboe
Quoted 44 lines · Jacob Thompson
Quoted 28 lines · Jens Axboe
Quoted 15 lines · Jacob Thompson
I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call
io_uring_setupandio_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with differentuser_data(0x1234+ i), and I used the opcodeIORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has theuser_dataas '18446744073709551615' which is correct, but the first 10 all hasuser_databe0x1234which is weird AF since only one item has thatuser_dataand I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter onceQuoted 11 lines
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615Quoted 8 lines
You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call
io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.Signature · 1 line
Jens AxboeI tried reproducing it in a smaller file. Assume I did everything wrong but somehow I seem to get results and they're not correct.
The codebase I'd like to use this in has very little activity (could go seconds without a single syscall), then execute a few hundreds-thousand (which I like to be async). SQPOLL sounds like the one best for my usecase. You can see I updated the sq tail before enter and I used
IORING_ENTER_SQ_WAKEUP+ slept for a second. The sq tail isn't zero which means I have results? and you can see its 10 of the sameuser_dataQuoted 12 lines
cq head is 0 enter result was 10
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
FFFFFFFF -1I looked at your test code, and you're setting up 10 NOP requests with userdata ==
0x1234, and hence you get 10 completions with that userdata. For some reason you iterate 11 CQEs, which means your last one is the one that you already filled with -1.In other words, it very much looks like it's working as it should. Any reason why you're using the raw interface rather than liburing? All of this seems to be not understanding how the ring works, and liburing helps isolate you from that. The SQ ring doesn't tell you anything about whether you have results (CQEs?), the difference between the SQ head and tail just tell you if there's something to submit. The CQ ring head and tail would tell you if there are CQEs to reap or not.
Quoted 7 lines
Quoted 2 lines
Signature · 1 line
Jens AxboeYou must be seeing something that I'm not. I had a +i in the line, should the
user_datanot increment every item? The line was 'sqes[i].user_data = 0x1234+i;'. The 11th iteration is intentional to see the value of the memset earlier.
You're not using IORING_SETUP_NO_SQARRAY, hence it's submitting index 0 every time. In other words, you're submitting the same SQE 10 times, not 10 different SQEs. That then yields 10 completions for an SQE with the same userdata, and hence your CQEs all look identical.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 81 lines · Jens Axboe
Quoted 72 lines · Jacob Thompson
Quoted 71 lines · Jens Axboe
Quoted 44 lines · Jacob Thompson
Quoted 28 lines · Jens Axboe
Quoted 15 lines · Jacob Thompson
I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call
io_uring_setupandio_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with differentuser_data(0x1234+ i), and I used the opcodeIORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has theuser_dataas '18446744073709551615' which is correct, but the first 10 all hasuser_databe0x1234which is weird AF since only one item has thatuser_dataand I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter onceQuoted 11 lines
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615Quoted 8 lines
You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call
io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.Signature · 1 line
Jens AxboeI tried reproducing it in a smaller file. Assume I did everything wrong but somehow I seem to get results and they're not correct.
The codebase I'd like to use this in has very little activity (could go seconds without a single syscall), then execute a few hundreds-thousand (which I like to be async). SQPOLL sounds like the one best for my usecase. You can see I updated the sq tail before enter and I used
IORING_ENTER_SQ_WAKEUP+ slept for a second. The sq tail isn't zero which means I have results? and you can see its 10 of the sameuser_dataQuoted 12 lines
cq head is 0 enter result was 10
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
FFFFFFFF -1I looked at your test code, and you're setting up 10 NOP requests with userdata ==
0x1234, and hence you get 10 completions with that userdata. For some reason you iterate 11 CQEs, which means your last one is the one that you already filled with -1.In other words, it very much looks like it's working as it should. Any reason why you're using the raw interface rather than liburing? All of this seems to be not understanding how the ring works, and liburing helps isolate you from that. The SQ ring doesn't tell you anything about whether you have results (CQEs?), the difference between the SQ head and tail just tell you if there's something to submit. The CQ ring head and tail would tell you if there are CQEs to reap or not.
Quoted 7 lines
Quoted 2 lines
Signature · 1 line
Jens AxboeYou must be seeing something that I'm not. I had a +i in the line, should the
user_datanot increment every item? The line was 'sqes[i].user_data = 0x1234+i;'. The 11th iteration is intentional to see the value of the memset earlier.
Quoted 4 lines
You're not using
IORING_SETUP_NO_SQARRAY, hence it's submitting index 0 every time. In other words, you're submitting the same SQE 10 times, not 10 different SQEs. That then yields 10 completions for an SQE with the same userdata, and hence your CQEs all look identical.
Ala the attached.
Signature · 1 line
Jens Axboe
#include <unistd.h>
#include <stdio.h>
#include <assert.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/syscall.h>
#include <linux/io_uring.h>
int io_uring_setup(unsigned entries, io_uring_params*params) { return syscall(__NR_io_uring_setup, entries, params); }
int io_uring_enter(int ring_fd, unsigned int to_submit, unsigned int min_complete, unsigned int flags, void*sig) { return (int) syscall(__NR_io_uring_enter, ring_fd, to_submit, min_complete, flags, 0); }
typedef int* IntPtr;
#define X(NAME) NAME = (int*)(p+params.sq_off.NAME) struct sqinfo
{
IntPtr head, tail, ring_mask, ring_entries, flags, dropped;
int*array;
void set(char*p, io_uring_params¶ms) { X(head); X(tail); X(ring_mask); X(ring_entries); X(flags); X(dropped); array = (int*)(p+params.sq_off.array); }
};
#undef X
#define X(NAME) NAME = (int*)(p+params.cq_off.NAME)
struct cqinfo
{
IntPtr head, tail, ring_mask, ring_entries, overflow, flags;
io_uring_cqe*cqes;
void set(char*p, io_uring_params¶ms) { X(head); X(tail); X(ring_mask); X(ring_entries); X(overflow); X(flags); cqes = (io_uring_cqe*)(p+params.cq_off.cqes); }
};#undef X
int main(int argc, char*argv[])
{
int queue_size = 256;
io_uring_params param{}; // zero init
param.flags = IORING_SETUP_NO_SQARRAY;
int ringFD = io_uring_setup(queue_size, ¶m);
assert(ringFD>0);
assert(param.features & IORING_FEAT_SINGLE_MMAP);
auto base_length = param.sq_off.array + param.sq_entries*4;
char *base = (char*) mmap(0, base_length, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, ringFD, IORING_OFF_SQ_RING);
assert(base != MAP_FAILED);
auto sqes = (io_uring_sqe*)mmap(0, param.sq_entries * sizeof(io_uring_sqe), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, ringFD, IORING_OFF_SQES);
assert(sqes != MAP_FAILED);
unsigned tail;
cqinfo cq;
sqinfo sq;
cq.set(base, param);
sq.set(base, param);
// Take 10 items
assert(*sq.tail == 0);
for(int i=0; i<10; i++) {
memset(&sqes[i], 0, sizeof(struct io_uring_sqe));
sqes[i].opcode = IORING_OP_NOP;
sqes[i].user_data = 0x1234+i;
}
__atomic_store_n(sq.tail, 10, __ATOMIC_RELEASE);
//int res = io_uring_enter(ringFD, 10, 10, IORING_ENTER_SQ_WAIT, 0);
int res = io_uring_enter(ringFD, 10, 10, IORING_ENTER_SQ_WAKEUP, 0);
sleep(1);
tail = __atomic_load_n(cq.tail, __ATOMIC_ACQUIRE);
printf("cq tail is %d enter result was %d\n", tail, res);
for(int i=0; i<tail; i++) {
printf("%X %d\n", cq.cqes[i].user_data, cq.cqes[i].res);
}
return 0;
}Quoted 81 lines · Jens Axboe
Quoted 72 lines · Jacob Thompson
Quoted 71 lines · Jens Axboe
Quoted 44 lines · Jacob Thompson
Quoted 28 lines · Jens Axboe
Quoted 15 lines · Jacob Thompson
I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call
io_uring_setupandio_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with differentuser_data(0x1234+ i), and I used the opcodeIORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has theuser_dataas '18446744073709551615' which is correct, but the first 10 all hasuser_databe0x1234which is weird AF since only one item has thatuser_dataand I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter onceQuoted 11 lines
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615Quoted 8 lines
You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call
io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.Signature · 1 line
Jens AxboeI tried reproducing it in a smaller file. Assume I did everything wrong but somehow I seem to get results and they're not correct.
The codebase I'd like to use this in has very little activity (could go seconds without a single syscall), then execute a few hundreds-thousand (which I like to be async). SQPOLL sounds like the one best for my usecase. You can see I updated the sq tail before enter and I used
IORING_ENTER_SQ_WAKEUP+ slept for a second. The sq tail isn't zero which means I have results? and you can see its 10 of the sameuser_dataQuoted 12 lines
cq head is 0 enter result was 10
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
FFFFFFFF -1I looked at your test code, and you're setting up 10 NOP requests with userdata ==
0x1234, and hence you get 10 completions with that userdata. For some reason you iterate 11 CQEs, which means your last one is the one that you already filled with -1.In other words, it very much looks like it's working as it should. Any reason why you're using the raw interface rather than liburing? All of this seems to be not understanding how the ring works, and liburing helps isolate you from that. The SQ ring doesn't tell you anything about whether you have results (CQEs?), the difference between the SQ head and tail just tell you if there's something to submit. The CQ ring head and tail would tell you if there are CQEs to reap or not.
Quoted 7 lines
Quoted 2 lines
Signature · 1 line
Jens AxboeYou must be seeing something that I'm not. I had a +i in the line, should the
user_datanot increment every item? The line was 'sqes[i].user_data = 0x1234+i;'. The 11th iteration is intentional to see the value of the memset earlier.
Quoted 7 lines
You're not using
IORING_SETUP_NO_SQARRAY, hence it's submitting index 0 every time. In other words, you're submitting the same SQE 10 times, not 10 different SQEs. That then yields 10 completions for an SQE with the same userdata, and hence your CQEs all look identical.Signature · 1 line
Jens Axboe
Thank you! When I read IORING_SETUP_NO_SQARRAY it went over my head and I thought it'd make sense later. I had no idea it would repeat 0 ten times when I call enter once, that's counter-intuitive. A lot of examples I'm going through uses all sorts of flags that I'm not familiar with. Is there something other than the man pages that I should read?
I saw IORING_SETUP_NO_SQARRAY in the lib and didn't think to turn on flags that I didn't understand.
Jens Axboe
maintainerQuoted 87 lines · Jacob Thompson
Quoted 86 lines · Jens Axboe
Quoted 72 lines · Jacob Thompson
Quoted 71 lines · Jens Axboe
Quoted 44 lines · Jacob Thompson
Quoted 28 lines · Jens Axboe
Quoted 15 lines · Jacob Thompson
I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call
io_uring_setupandio_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with differentuser_data(0x1234+ i), and I used the opcodeIORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has theuser_dataas '18446744073709551615' which is correct, but the first 10 all hasuser_databe0x1234which is weird AF since only one item has thatuser_dataand I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter onceQuoted 11 lines
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615Quoted 8 lines
You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call
io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.Signature · 1 line
Jens AxboeI tried reproducing it in a smaller file. Assume I did everything wrong but somehow I seem to get results and they're not correct.
The codebase I'd like to use this in has very little activity (could go seconds without a single syscall), then execute a few hundreds-thousand (which I like to be async). SQPOLL sounds like the one best for my usecase. You can see I updated the sq tail before enter and I used
IORING_ENTER_SQ_WAKEUP+ slept for a second. The sq tail isn't zero which means I have results? and you can see its 10 of the sameuser_dataQuoted 12 lines
cq head is 0 enter result was 10
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
FFFFFFFF -1I looked at your test code, and you're setting up 10 NOP requests with userdata ==
0x1234, and hence you get 10 completions with that userdata. For some reason you iterate 11 CQEs, which means your last one is the one that you already filled with -1.In other words, it very much looks like it's working as it should. Any reason why you're using the raw interface rather than liburing? All of this seems to be not understanding how the ring works, and liburing helps isolate you from that. The SQ ring doesn't tell you anything about whether you have results (CQEs?), the difference between the SQ head and tail just tell you if there's something to submit. The CQ ring head and tail would tell you if there are CQEs to reap or not.
Quoted 7 lines
Quoted 2 lines
Signature · 1 line
Jens AxboeYou must be seeing something that I'm not. I had a +i in the line, should the
user_datanot increment every item? The line was 'sqes[i].user_data = 0x1234+i;'. The 11th iteration is intentional to see the value of the memset earlier.You're not using
IORING_SETUP_NO_SQARRAY, hence it's submitting index 0 every time. In other words, you're submitting the same SQE 10 times, not 10 different SQEs. That then yields 10 completions for an SQE with the same userdata, and hence your CQEs all look identical.
Quoted 6 lines
Quoted 2 lines
Signature · 1 line
Jens AxboeThank you! When I read
IORING_SETUP_NO_SQARRAYit went over my head and I thought it'd make sense later. I had no idea it would repeat 0 ten times when I call enter once, that's counter-intuitive.
It's not repeating 0 ten times, it's iterating the sqe indirection array from 0..9, but each entry contains 0 as that is what it was memset too. Hence it'll submit sqes[0] ten times. If you don't use IORING_SETUP_NO_SQARRAY, then there's an indirection where when you ask it to submit 10 requests and the SQ head is currently at 0, it'll index sq->array[0..9] to find the sqes offset it should submit from. If you use IORING_SETUP_NO_SQARRAY, then no such indirection exists, and when you ask it to submit 0..9 then that'll be the direct offset from the SQ head into the sqes array.
Quoted 4 lines
A lot of examples I'm going through uses all sorts of flags that I'm not familiar with. Is there something other than the man pages that I should read? I saw
IORING_SETUP_NO_SQARRAYin the lib and didn't think to turn on flags that I didn't understand.
io_uring_setup.2 should have all of these setup flags documented.
I'll ask again since you didn't answer - why aren't you just using liburing? It'll handle all of these details for you, so you can focus on just using the commands you need and not need to build your own abstraction around how to handle the SQ and CQ rings manually.
Signature · 1 line
Jens Axboe
Quoted 9 lines · Jens Axboe
io_uring_setup.2 should have all of these setup flags documented.
I'll ask again since you didn't answer - why aren't you just using liburing? It'll handle all of these details for you, so you can focus on just using the commands you need and not need to build your own abstraction around how to handle the SQ and CQ rings manually.
Signature · 1 line
Jens Axboe
I read through liburing, boss wants me to understand everything about io_uring and wants an in house solution and for me to optimize it. I was able to write a test that submitted more than 2^32 entries and it ran correctly. I think I'm off to a good start now
Jens Axboe
maintainerQuoted 8 lines · Jacob Thompson
Quoted 7 lines · Jens Axboe
io_uring_setup.2 should have all of these setup flags documented.
I'll ask again since you didn't answer - why aren't you just using liburing? It'll handle all of these details for you, so you can focus on just using the commands you need and not need to build your own abstraction around how to handle the SQ and CQ rings manually.
Quoted 5 lines
Quoted 2 lines
Signature · 1 line
Jens AxboeI read through liburing, boss wants me to understand everything about
io_uringand wants an in house solution and for me to optimize it.
Ok I take it back, boss sounds reasonable then ;-)
Quoted 2 lines
I was able to write a test that submitted more than 2^32 entries and it ran correctly. I think I'm off to a good start now
Good!
Signature · 1 line
Jens Axboe
I made two typos in my last email. The cq *tail* was 10, and I meant to say the cq tail* isn't zero which means I have results? I'm planning to reread the man pages but I feel like that's trying to find a solution in a haystack
Jens Axboe
maintainerQuoted 4 lines · Jacob Thompson
I made two typos in my last email. The cq *tail* was 10, and I meant to say the cq tail* isn't zero which means I have results? I'm planning to reread the man pages but I feel like that's trying to find a solution in a haystack
I'd encourage you to read and understand liburing, even if you aren't going to use it. That will tell you how to reap and iterate CQE events.
Signature · 1 line
Jens Axboe