Skip to contentSkip to conversation

CQE repeats the first item?

DISCUSSION Jacob Thompson 2 participants 13 messages Oct 5, 2025 – Oct 6, 2025 io_uring

lore
Share & tools
lore
#0 lore raw

I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description
I'm using syscalls to call io_uring_setup and io_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again.
In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with different user_data (0x1234 + i), and I used the opcode IORING_OP_NOP.
I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has the user_data as '18446744073709551615' which is correct, but the first 10 all has user_data be 0x1234 which is weird AF since only one item has that user_data and I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter once

Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615

maintainer
#1 lore raw
Quoted 15 lines · Jacob Thompson

I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call io_uring_setup and io_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with different user_data (0x1234 + i), and I used the opcode IORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has the user_data as '18446744073709551615' which is correct, but the first 10 all has user_data be 0x1234 which is weird AF since only one item has that user_data and I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter once

Quoted 11 lines

Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615

You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.

Signature · 1 line
Jens Axboe
#2 lore raw
Quoted 28 lines · Jens Axboe
Quoted 15 lines · Jacob Thompson

I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call io_uring_setup and io_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with different user_data (0x1234 + i), and I used the opcode IORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has the user_data as '18446744073709551615' which is correct, but the first 10 all has user_data be 0x1234 which is weird AF since only one item has that user_data and I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter once

Quoted 11 lines

Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615

Quoted 8 lines

You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.

Signature · 1 line
Jens Axboe

I tried reproducing it in a smaller file. Assume I did everything wrong but somehow I seem to get results and they're not correct.

The codebase I'd like to use this in has very little activity (could go seconds without a single syscall), then execute a few hundreds-thousand (which I like to be async). SQPOLL sounds like the one best for my usecase. You can see I updated the sq tail before enter and I used IORING_ENTER_SQ_WAKEUP + slept for a second. The sq tail isn't zero which means I have results? and you can see its 10 of the same user_data

cq head is 0 enter result was 10
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
FFFFFFFF -1

#include <unistd.h>
#include <stdio.h>
#include <assert.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/syscall.h>
#include <linux/io_uring.h>

int io_uring_setup(unsigned entries, io_uring_params*params) { return syscall(__NR_io_uring_setup, entries, params); }
int io_uring_enter(int ring_fd, unsigned int to_submit, unsigned int min_complete, unsigned int flags, void*sig) { return (int) syscall(__NR_io_uring_enter, ring_fd, to_submit, min_complete, flags, 0); }

typedef int* IntPtr;
#define X(NAME) NAME = (int*)(p+params.sq_off.NAME) struct sqinfo

{
	IntPtr head, tail, ring_mask, ring_entries, flags, dropped;
	int*array;
	void set(char*p, io_uring_params&params) { X(head); X(tail); X(ring_mask); X(ring_entries); X(flags); X(dropped); array = (int*)(p+params.sq_off.array); }
};
#undef X
#define X(NAME) NAME = (int*)(p+params.cq_off.NAME)
struct cqinfo
{
	IntPtr head, tail, ring_mask, ring_entries, overflow, flags;
	io_uring_cqe*cqes;
	void set(char*p, io_uring_params&params) { X(head); X(tail); X(ring_mask); X(ring_entries); X(overflow); X(flags); cqes = (io_uring_cqe*)(p+params.cq_off.cqes); }
};

#undef X

int main(int argc, char*argv[])

{
	int queue_size = 256;
	io_uring_params param{}; // zero init
	param.flags = IORING_SETUP_SQPOLL;
	int ringFD = io_uring_setup(queue_size, &param);
	assert(ringFD>0);
	assert(param.features & IORING_FEAT_SINGLE_MMAP);
	auto base_length = param.sq_off.array + param.sq_entries*4;
	char *base = (char*) mmap(0, base_length, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, ringFD, IORING_OFF_SQ_RING);
	assert(base != MAP_FAILED);
	auto sqes = (io_uring_sqe*)mmap(0, param.sq_entries * sizeof(io_uring_sqe), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, ringFD, IORING_OFF_SQES);
	assert(sqes != MAP_FAILED);

	cqinfo cq;
	sqinfo sq;

	cq.set(base, param);
	sq.set(base, param);
	for(int i=0; i<*sq.ring_entries; i++) {
		memset(sqes+i, -1, sizeof(io_uring_sqe));
	}
	for(int i=0; i<*cq.ring_entries; i++) {
		memset(cq.cqes+i, -1, sizeof(io_uring_cqe));
	}


	// Take 10 items
	assert(*sq.tail == 0);
	auto p = &sqes[0];
	memset(p, 0, sizeof(io_uring_sqe)*10);
	for(int i=0; i<10; i++) {
		sqes[i].opcode = IORING_OP_NOP;
		sqes[i].user_data = 0x1234+i;
	}

	__atomic_store_n(sq.tail, 10, __ATOMIC_RELEASE);

	//int res = io_uring_enter(ringFD, 10, 10, IORING_ENTER_SQ_WAIT, 0);
	int res = io_uring_enter(ringFD, 10, 10, IORING_ENTER_SQ_WAKEUP, 0);

	sleep(1);
	printf("cq tail is %d enter result was %d\n", __atomic_load_n(cq.tail, __ATOMIC_ACQUIRE), res);

	for(int i=0; i<11; i++) {
		printf("%X %d\n", cq.cqes[i].user_data, cq.cqes[i].res);
	}
	return 0;
}
maintainer
#3 lore raw
Quoted 44 lines · Jacob Thompson
Quoted 28 lines · Jens Axboe
Quoted 15 lines · Jacob Thompson

I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call io_uring_setup and io_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with different user_data (0x1234 + i), and I used the opcode IORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has the user_data as '18446744073709551615' which is correct, but the first 10 all has user_data be 0x1234 which is weird AF since only one item has that user_data and I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter once

Quoted 11 lines

Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615

Quoted 8 lines

You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.

Signature · 1 line
Jens Axboe

I tried reproducing it in a smaller file. Assume I did everything wrong but somehow I seem to get results and they're not correct.

The codebase I'd like to use this in has very little activity (could go seconds without a single syscall), then execute a few hundreds-thousand (which I like to be async). SQPOLL sounds like the one best for my usecase. You can see I updated the sq tail before enter and I used IORING_ENTER_SQ_WAKEUP + slept for a second. The sq tail isn't zero which means I have results? and you can see its 10 of the same user_data

Quoted 12 lines

cq head is 0 enter result was 10
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
FFFFFFFF -1

I looked at your test code, and you're setting up 10 NOP requests with userdata == 0x1234, and hence you get 10 completions with that userdata. For some reason you iterate 11 CQEs, which means your last one is the one that you already filled with -1.

In other words, it very much looks like it's working as it should. Any reason why you're using the raw interface rather than liburing? All of this seems to be not understanding how the ring works, and liburing helps isolate you from that. The SQ ring doesn't tell you anything about whether you have results (CQEs?), the difference between the SQ head and tail just tell you if there's something to submit. The CQ ring head and tail would tell you if there are CQEs to reap or not.

Signature · 1 line
Jens Axboe
#4 lore raw
Quoted 63 lines · Jens Axboe
Quoted 44 lines · Jacob Thompson
Quoted 28 lines · Jens Axboe
Quoted 15 lines · Jacob Thompson

I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call io_uring_setup and io_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with different user_data (0x1234 + i), and I used the opcode IORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has the user_data as '18446744073709551615' which is correct, but the first 10 all has user_data be 0x1234 which is weird AF since only one item has that user_data and I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter once

Quoted 11 lines

Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615

Quoted 8 lines

You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.

Signature · 1 line
Jens Axboe

I tried reproducing it in a smaller file. Assume I did everything wrong but somehow I seem to get results and they're not correct.

The codebase I'd like to use this in has very little activity (could go seconds without a single syscall), then execute a few hundreds-thousand (which I like to be async). SQPOLL sounds like the one best for my usecase. You can see I updated the sq tail before enter and I used IORING_ENTER_SQ_WAKEUP + slept for a second. The sq tail isn't zero which means I have results? and you can see its 10 of the same user_data

Quoted 12 lines

cq head is 0 enter result was 10
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
FFFFFFFF -1

I looked at your test code, and you're setting up 10 NOP requests with userdata == 0x1234, and hence you get 10 completions with that userdata. For some reason you iterate 11 CQEs, which means your last one is the one that you already filled with -1.

Quoted 10 lines

In other words, it very much looks like it's working as it should. Any reason why you're using the raw interface rather than liburing? All of this seems to be not understanding how the ring works, and liburing helps isolate you from that. The SQ ring doesn't tell you anything about whether you have results (CQEs?), the difference between the SQ head and tail just tell you if there's something to submit. The CQ ring head and tail would tell you if there are CQEs to reap or not.

Signature · 1 line
Jens Axboe

You must be seeing something that I'm not. I had a +i in the line, should the user_data not increment every item? The line was 'sqes[i].user_data = 0x1234+i;'. The 11th iteration is intentional to see the value of the memset earlier.

Quoted 1 line

I'd encourage you to read and understand liburing, even if you aren't going to use it. That will tell you how to reap and iterate CQE events.

I did. I didn't try to compile and step through but I'm in the middle of rereading the manual. I want to finish it before I look at the lib again

maintainer
#5 lore raw
Quoted 72 lines · Jacob Thompson
Quoted 71 lines · Jens Axboe
Quoted 44 lines · Jacob Thompson
Quoted 28 lines · Jens Axboe
Quoted 15 lines · Jacob Thompson

I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call io_uring_setup and io_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with different user_data (0x1234 + i), and I used the opcode IORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has the user_data as '18446744073709551615' which is correct, but the first 10 all has user_data be 0x1234 which is weird AF since only one item has that user_data and I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter once

Quoted 11 lines

Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615

Quoted 8 lines

You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.

Signature · 1 line
Jens Axboe

I tried reproducing it in a smaller file. Assume I did everything wrong but somehow I seem to get results and they're not correct.

The codebase I'd like to use this in has very little activity (could go seconds without a single syscall), then execute a few hundreds-thousand (which I like to be async). SQPOLL sounds like the one best for my usecase. You can see I updated the sq tail before enter and I used IORING_ENTER_SQ_WAKEUP + slept for a second. The sq tail isn't zero which means I have results? and you can see its 10 of the same user_data

Quoted 12 lines

cq head is 0 enter result was 10
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
FFFFFFFF -1

I looked at your test code, and you're setting up 10 NOP requests with userdata == 0x1234, and hence you get 10 completions with that userdata. For some reason you iterate 11 CQEs, which means your last one is the one that you already filled with -1.

In other words, it very much looks like it's working as it should. Any reason why you're using the raw interface rather than liburing? All of this seems to be not understanding how the ring works, and liburing helps isolate you from that. The SQ ring doesn't tell you anything about whether you have results (CQEs?), the difference between the SQ head and tail just tell you if there's something to submit. The CQ ring head and tail would tell you if there are CQEs to reap or not.

Quoted 7 lines
Quoted 2 lines
Signature · 1 line
Jens Axboe

You must be seeing something that I'm not. I had a +i in the line, should the user_data not increment every item? The line was 'sqes[i].user_data = 0x1234+i;'. The 11th iteration is intentional to see the value of the memset earlier.

You're not using IORING_SETUP_NO_SQARRAY, hence it's submitting index 0 every time. In other words, you're submitting the same SQE 10 times, not 10 different SQEs. That then yields 10 completions for an SQE with the same userdata, and hence your CQEs all look identical.

Signature · 1 line
Jens Axboe
maintainer
#6 lore raw
Quoted 81 lines · Jens Axboe
Quoted 72 lines · Jacob Thompson
Quoted 71 lines · Jens Axboe
Quoted 44 lines · Jacob Thompson
Quoted 28 lines · Jens Axboe
Quoted 15 lines · Jacob Thompson

I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call io_uring_setup and io_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with different user_data (0x1234 + i), and I used the opcode IORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has the user_data as '18446744073709551615' which is correct, but the first 10 all has user_data be 0x1234 which is weird AF since only one item has that user_data and I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter once

Quoted 11 lines

Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615

Quoted 8 lines

You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.

Signature · 1 line
Jens Axboe

I tried reproducing it in a smaller file. Assume I did everything wrong but somehow I seem to get results and they're not correct.

The codebase I'd like to use this in has very little activity (could go seconds without a single syscall), then execute a few hundreds-thousand (which I like to be async). SQPOLL sounds like the one best for my usecase. You can see I updated the sq tail before enter and I used IORING_ENTER_SQ_WAKEUP + slept for a second. The sq tail isn't zero which means I have results? and you can see its 10 of the same user_data

Quoted 12 lines

cq head is 0 enter result was 10
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
FFFFFFFF -1

I looked at your test code, and you're setting up 10 NOP requests with userdata == 0x1234, and hence you get 10 completions with that userdata. For some reason you iterate 11 CQEs, which means your last one is the one that you already filled with -1.

In other words, it very much looks like it's working as it should. Any reason why you're using the raw interface rather than liburing? All of this seems to be not understanding how the ring works, and liburing helps isolate you from that. The SQ ring doesn't tell you anything about whether you have results (CQEs?), the difference between the SQ head and tail just tell you if there's something to submit. The CQ ring head and tail would tell you if there are CQEs to reap or not.

Quoted 7 lines
Quoted 2 lines
Signature · 1 line
Jens Axboe

You must be seeing something that I'm not. I had a +i in the line, should the user_data not increment every item? The line was 'sqes[i].user_data = 0x1234+i;'. The 11th iteration is intentional to see the value of the memset earlier.

Quoted 4 lines

You're not using IORING_SETUP_NO_SQARRAY, hence it's submitting index 0 every time. In other words, you're submitting the same SQE 10 times, not 10 different SQEs. That then yields 10 completions for an SQE with the same userdata, and hence your CQEs all look identical.

Ala the attached.

Signature · 1 line
Jens Axboe

#include <unistd.h>
#include <stdio.h>
#include <assert.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/syscall.h>
#include <linux/io_uring.h>

int io_uring_setup(unsigned entries, io_uring_params*params) { return syscall(__NR_io_uring_setup, entries, params); }
int io_uring_enter(int ring_fd, unsigned int to_submit, unsigned int min_complete, unsigned int flags, void*sig) { return (int) syscall(__NR_io_uring_enter, ring_fd, to_submit, min_complete, flags, 0); }

typedef int* IntPtr;
#define X(NAME) NAME = (int*)(p+params.sq_off.NAME) struct sqinfo

{
	IntPtr head, tail, ring_mask, ring_entries, flags, dropped;
	int*array;
	void set(char*p, io_uring_params&params) { X(head); X(tail); X(ring_mask); X(ring_entries); X(flags); X(dropped); array = (int*)(p+params.sq_off.array); }
};
#undef X
#define X(NAME) NAME = (int*)(p+params.cq_off.NAME)
struct cqinfo
{
	IntPtr head, tail, ring_mask, ring_entries, overflow, flags;
	io_uring_cqe*cqes;
	void set(char*p, io_uring_params&params) { X(head); X(tail); X(ring_mask); X(ring_entries); X(overflow); X(flags); cqes = (io_uring_cqe*)(p+params.cq_off.cqes); }
};

#undef X

int main(int argc, char*argv[])

{
	int queue_size = 256;
	io_uring_params param{}; // zero init
	param.flags = IORING_SETUP_NO_SQARRAY;
	int ringFD = io_uring_setup(queue_size, &param);
	assert(ringFD>0);
	assert(param.features & IORING_FEAT_SINGLE_MMAP);
	auto base_length = param.sq_off.array + param.sq_entries*4;
	char *base = (char*) mmap(0, base_length, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, ringFD, IORING_OFF_SQ_RING);
	assert(base != MAP_FAILED);
	auto sqes = (io_uring_sqe*)mmap(0, param.sq_entries * sizeof(io_uring_sqe), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, ringFD, IORING_OFF_SQES);
	assert(sqes != MAP_FAILED);
	unsigned tail;

	cqinfo cq;
	sqinfo sq;

	cq.set(base, param);
	sq.set(base, param);

	// Take 10 items
	assert(*sq.tail == 0);
	for(int i=0; i<10; i++) {
		memset(&sqes[i], 0, sizeof(struct io_uring_sqe));
		sqes[i].opcode = IORING_OP_NOP;
		sqes[i].user_data = 0x1234+i;
	}

	__atomic_store_n(sq.tail, 10, __ATOMIC_RELEASE);

	//int res = io_uring_enter(ringFD, 10, 10, IORING_ENTER_SQ_WAIT, 0);
	int res = io_uring_enter(ringFD, 10, 10, IORING_ENTER_SQ_WAKEUP, 0);

	sleep(1);
	tail = __atomic_load_n(cq.tail, __ATOMIC_ACQUIRE);
	printf("cq tail is %d enter result was %d\n", tail, res);

	for(int i=0; i<tail; i++) {
		printf("%X %d\n", cq.cqes[i].user_data, cq.cqes[i].res);
	}
	return 0;
}
#7 lore raw
In reply to Jens Axboe
Quoted 81 lines · Jens Axboe
Quoted 72 lines · Jacob Thompson
Quoted 71 lines · Jens Axboe
Quoted 44 lines · Jacob Thompson
Quoted 28 lines · Jens Axboe
Quoted 15 lines · Jacob Thompson

I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call io_uring_setup and io_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with different user_data (0x1234 + i), and I used the opcode IORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has the user_data as '18446744073709551615' which is correct, but the first 10 all has user_data be 0x1234 which is weird AF since only one item has that user_data and I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter once

Quoted 11 lines

Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615

Quoted 8 lines

You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.

Signature · 1 line
Jens Axboe

I tried reproducing it in a smaller file. Assume I did everything wrong but somehow I seem to get results and they're not correct.

The codebase I'd like to use this in has very little activity (could go seconds without a single syscall), then execute a few hundreds-thousand (which I like to be async). SQPOLL sounds like the one best for my usecase. You can see I updated the sq tail before enter and I used IORING_ENTER_SQ_WAKEUP + slept for a second. The sq tail isn't zero which means I have results? and you can see its 10 of the same user_data

Quoted 12 lines

cq head is 0 enter result was 10
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
FFFFFFFF -1

I looked at your test code, and you're setting up 10 NOP requests with userdata == 0x1234, and hence you get 10 completions with that userdata. For some reason you iterate 11 CQEs, which means your last one is the one that you already filled with -1.

In other words, it very much looks like it's working as it should. Any reason why you're using the raw interface rather than liburing? All of this seems to be not understanding how the ring works, and liburing helps isolate you from that. The SQ ring doesn't tell you anything about whether you have results (CQEs?), the difference between the SQ head and tail just tell you if there's something to submit. The CQ ring head and tail would tell you if there are CQEs to reap or not.

Quoted 7 lines
Quoted 2 lines
Signature · 1 line
Jens Axboe

You must be seeing something that I'm not. I had a +i in the line, should the user_data not increment every item? The line was 'sqes[i].user_data = 0x1234+i;'. The 11th iteration is intentional to see the value of the memset earlier.

Quoted 7 lines

You're not using IORING_SETUP_NO_SQARRAY, hence it's submitting index 0 every time. In other words, you're submitting the same SQE 10 times, not 10 different SQEs. That then yields 10 completions for an SQE with the same userdata, and hence your CQEs all look identical.

Signature · 1 line
Jens Axboe

Thank you! When I read IORING_SETUP_NO_SQARRAY it went over my head and I thought it'd make sense later. I had no idea it would repeat 0 ten times when I call enter once, that's counter-intuitive. A lot of examples I'm going through uses all sorts of flags that I'm not familiar with. Is there something other than the man pages that I should read?
I saw IORING_SETUP_NO_SQARRAY in the lib and didn't think to turn on flags that I didn't understand.

maintainer
#8 lore raw
Quoted 87 lines · Jacob Thompson
Quoted 86 lines · Jens Axboe
Quoted 72 lines · Jacob Thompson
Quoted 71 lines · Jens Axboe
Quoted 44 lines · Jacob Thompson
Quoted 28 lines · Jens Axboe
Quoted 15 lines · Jacob Thompson

I'm doing something wrong and I wanted to know if anyone knows what I did wrong from the description I'm using syscalls to call io_uring_setup and io_uring_enter. I managed to submit 1 item without an issue but any more gets me the first item over and over again. In my test I did a memset -1 on cqes and sqes, I memset 0 the first ten sqes with different user_data (0x1234 + i), and I used the opcode IORING_OP_NOP. I called "io_uring_enter(fd, 10, 0, IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has the user_data as '18446744073709551615' which is correct, but the first 10 all has user_data be 0x1234 which is weird AF since only one item has that user_data and I submited 10 I considered maybe the debugger was giving me incorrect values so I tried printing the user data in a loop, I have no idea why the first one repeats 10 times. I only called enter once

Quoted 11 lines

Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 4660
Id is 18446744073709551615

Quoted 8 lines

You're presumably not updating your side of the CQ ring correctly, see what liburing does when you call io_uring_cqe_seen(). If that's not it, then you're probably mishandling something else and an example would be useful as otherwise I'd just be guessing. There's really not much to go from in this report.

Signature · 1 line
Jens Axboe

I tried reproducing it in a smaller file. Assume I did everything wrong but somehow I seem to get results and they're not correct.

The codebase I'd like to use this in has very little activity (could go seconds without a single syscall), then execute a few hundreds-thousand (which I like to be async). SQPOLL sounds like the one best for my usecase. You can see I updated the sq tail before enter and I used IORING_ENTER_SQ_WAKEUP + slept for a second. The sq tail isn't zero which means I have results? and you can see its 10 of the same user_data

Quoted 12 lines

cq head is 0 enter result was 10
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
1234 0
FFFFFFFF -1

I looked at your test code, and you're setting up 10 NOP requests with userdata == 0x1234, and hence you get 10 completions with that userdata. For some reason you iterate 11 CQEs, which means your last one is the one that you already filled with -1.

In other words, it very much looks like it's working as it should. Any reason why you're using the raw interface rather than liburing? All of this seems to be not understanding how the ring works, and liburing helps isolate you from that. The SQ ring doesn't tell you anything about whether you have results (CQEs?), the difference between the SQ head and tail just tell you if there's something to submit. The CQ ring head and tail would tell you if there are CQEs to reap or not.

Quoted 7 lines
Quoted 2 lines
Signature · 1 line
Jens Axboe

You must be seeing something that I'm not. I had a +i in the line, should the user_data not increment every item? The line was 'sqes[i].user_data = 0x1234+i;'. The 11th iteration is intentional to see the value of the memset earlier.

You're not using IORING_SETUP_NO_SQARRAY, hence it's submitting index 0 every time. In other words, you're submitting the same SQE 10 times, not 10 different SQEs. That then yields 10 completions for an SQE with the same userdata, and hence your CQEs all look identical.

Quoted 6 lines
Quoted 2 lines
Signature · 1 line
Jens Axboe

Thank you! When I read IORING_SETUP_NO_SQARRAY it went over my head and I thought it'd make sense later. I had no idea it would repeat 0 ten times when I call enter once, that's counter-intuitive.

It's not repeating 0 ten times, it's iterating the sqe indirection array from 0..9, but each entry contains 0 as that is what it was memset too. Hence it'll submit sqes[0] ten times. If you don't use IORING_SETUP_NO_SQARRAY, then there's an indirection where when you ask it to submit 10 requests and the SQ head is currently at 0, it'll index sq->array[0..9] to find the sqes offset it should submit from. If you use IORING_SETUP_NO_SQARRAY, then no such indirection exists, and when you ask it to submit 0..9 then that'll be the direct offset from the SQ head into the sqes array.

Quoted 4 lines

A lot of examples I'm going through uses all sorts of flags that I'm not familiar with. Is there something other than the man pages that I should read? I saw IORING_SETUP_NO_SQARRAY in the lib and didn't think to turn on flags that I didn't understand.

io_uring_setup.2 should have all of these setup flags documented.

I'll ask again since you didn't answer - why aren't you just using liburing? It'll handle all of these details for you, so you can focus on just using the commands you need and not need to build your own abstraction around how to handle the SQ and CQ rings manually.

Signature · 1 line
Jens Axboe
#9 lore raw
Quoted 9 lines · Jens Axboe

io_uring_setup.2 should have all of these setup flags documented.

I'll ask again since you didn't answer - why aren't you just using liburing? It'll handle all of these details for you, so you can focus on just using the commands you need and not need to build your own abstraction around how to handle the SQ and CQ rings manually.

Signature · 1 line
Jens Axboe

I read through liburing, boss wants me to understand everything about io_uring and wants an in house solution and for me to optimize it. I was able to write a test that submitted more than 2^32 entries and it ran correctly. I think I'm off to a good start now

maintainer
#10 lore raw
Quoted 8 lines · Jacob Thompson
Quoted 7 lines · Jens Axboe

io_uring_setup.2 should have all of these setup flags documented.

I'll ask again since you didn't answer - why aren't you just using liburing? It'll handle all of these details for you, so you can focus on just using the commands you need and not need to build your own abstraction around how to handle the SQ and CQ rings manually.

Quoted 5 lines
Quoted 2 lines
Signature · 1 line
Jens Axboe

I read through liburing, boss wants me to understand everything about io_uring and wants an in house solution and for me to optimize it.

Ok I take it back, boss sounds reasonable then ;-)

Quoted 2 lines

I was able to write a test that submitted more than 2^32 entries and it ran correctly. I think I'm off to a good start now

Good!

Signature · 1 line
Jens Axboe
#11 lore raw
In reply to Jens Axboe

I made two typos in my last email. The cq *tail* was 10, and I meant to say the cq tail* isn't zero which means I have results? I'm planning to reread the man pages but I feel like that's trying to find a solution in a haystack

maintainer
#12 lore raw
Quoted 4 lines · Jacob Thompson

I made two typos in my last email. The cq *tail* was 10, and I meant to say the cq tail* isn't zero which means I have results? I'm planning to reread the man pages but I feel like that's trying to find a solution in a haystack

I'd encourage you to read and understand liburing, even if you aren't going to use it. That will tell you how to reap and iterate CQE events.

Signature · 1 line
Jens Axboe