Skip to contentSkip to conversation

providers/rxe: Replace '%' with '&' in check_qp_queue_full()

PATCH [email protected] 2 participants 3 messages Jan 18, 2022 – Jan 21, 2022 InfiniBand

lore
Share & tools
lore
#0 lore raw
CC linux-rdma mail list
Quoted 2 lines · Bob Pearson

The way these queues work they can only hold 2^n - 1 entries. The reason for this is
to distinguish empty and full. If you let the last entry be written then producer would advance to equal consumer which is the initial empty condition. So a queue which can hold 1 entry has to have two slots (n=1) but can only hold one entry. It begins with

Hi Bob,

Thanks for your detailed example.

Quoted 8 lines

prod = cons = 0 and is empty and is*not* full

Adding one entry gives

slot[0] = value, prod = 1, cons = 0 and is full and not empty

After reading this value

Quoted 7 lines

slot[0] = old value, prod = cons = 1 and queue is empty again.

Writing a new value

slot[1] = new value, slot[0] = old value, prod = 0 and cons = 1 and queue is full again.

The expression full = (cons == ((qp->cur_index + 1) % q->index_mask)) is correct.

The issue happens here, let's look at the "writing a new value" step. We think queue is full again when prod = 0 and cons = 1, but this expression thinks the queue is not full.
cons == ((qp->cur_index + 1) % q->index_mask))
<1> == <0 + 1> % <1>

In addition, this expression thinks the queue is full wrongly when we create an empty two-slots queue (i.e. prod = 0 and cons = 0). cons == ((qp->cur_index + 1) % q->index_mask))
<0> == <0 + 1> % <1>

The following steps can expose the issue:
-------------------------------------------
attr_ex.cap.max_send_wr = attr_ex.cap.max_recv_wr = 1;
...
ret = rdma_create_qp_ex(id, &attr_ex);
...
ibv_wr_start(qpx);
ibv_wr_rdma_atomic_write(qpx,...); --> check_qp_queue_full() returns ENOSPC wrongly.
ibv_wr_complete(qpx);
-------------------------------------------
Please see my example[1] for details.

[1]
https://github.com/yangx-jy/rdma-core/commit/257c6672dc147e0881525e469a49d933f1bc9183

I think the commit message misled you.

Best Regards,
Xiao Yang

Quoted 3 lines

Please do not make this change.

Bob

#1 lore raw
Quoted 9 lines · On 1/18/22 03:02, [email protected] wrote:
CC linux-rdma mail list
Quoted 2 lines · Bob Pearson

The way these queues work they can only hold 2^n - 1 entries. The reason for this is
to distinguish empty and full. If you let the last entry be written then producer would advance to equal consumer which is the initial empty condition. So a queue which can hold 1 entry has to have two slots (n=1) but can only hold one entry. It begins with

Hi Bob,

Thanks for your detailed example.

Quoted 1 line

prod = cons = 0 and is empty and is*not* full

full = (cons == ((prod + 1) % 2)) = (0 == 1) = false empty = (prod == cons) == (0 == 0) = true

Quoted 4 lines
Quoted 4 lines

Adding one entry gives

slot[0] = value, prod = 1, cons = 0 and is full and not empty

full = (cons == ((prod + 1) % 2)) = (0 == 0) = true empty = (prod == cons) == (0 == 1) = false

Quoted 4 lines
Quoted 4 lines

After reading this value

slot[0] = old value, prod = cons = 1 and queue is empty again.

full = (cons == ((prod + 1) % 2)) = (1 == 0) = false empty = (prod == cons) = (1 == 1) = true

Quoted 4 lines
Quoted 4 lines

Writing a new value

slot[1] = new value, slot[0] = old value, prod = 0 and cons = 1 and queue is full again.

full = (cons == ((prod + 1) % 2)) = (1 == (0 + 1) % 2) = true empty = (prod == cons) = (0 == 1) = false

and x % 2^N is the same as x & (2^N - 1) or x & index_mask.

Quoted 2 lines
Quoted 2 lines

The expression full = (cons == ((qp->cur_index + 1) % q->index_mask)) is correct.

Actually you are correct. If you look at queue_full() it uses full = (cons == (prod + 1) & q->index_mask) and queue_empty() uses empty = (prod == cons) *but*
check_qp_queue_full() uses
full = (cons == ((qp->cur_index + 1) % q->index_mask)) with % instead of &. I was so used to looking at the first two I missed the error in the last one.

% should be & like the others.

Bob

#2 lore raw
Quoted 22 lines · Bob Pearson
Quoted 9 lines · On 1/18/22 03:02, [email protected] wrote:
CC linux-rdma mail list
Quoted 2 lines · Bob Pearson

The way these queues work they can only hold 2^n - 1 entries. The reason for this is
to distinguish empty and full. If you let the last entry be written then producer would advance to equal consumer which is the initial empty condition. So a queue which can hold 1 entry has to have two slots (n=1) but can only hold one entry. It begins with

Hi Bob,

Thanks for your detailed example.

Quoted 1 line

prod = cons = 0 and is empty and is*not* full

full = (cons == ((prod + 1) % 2)) = (0 == 1) = false empty = (prod == cons) == (0 == 0) = true

Quoted 3 lines
Quoted 3 lines

Adding one entry gives

slot[0] = value, prod = 1, cons = 0 and is full and not empty

full = (cons == ((prod + 1) % 2)) = (0 == 0) = true empty = (prod == cons) == (0 == 1) = false

Quoted 3 lines
Quoted 3 lines

After reading this value

slot[0] = old value, prod = cons = 1 and queue is empty again.

full = (cons == ((prod + 1) % 2)) = (1 == 0) = false empty = (prod == cons) = (1 == 1) = true

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

Writing a new value

slot[1] = new value, slot[0] = old value, prod = 0 and cons = 1 and queue is full again.

full = (cons == ((prod + 1) % 2)) = (1 == (0 + 1) % 2) = true empty = (prod == cons) = (0 == 1) = false

and x % 2^N is the same as x& (2^N - 1) or x& index_mask.

Hi Bob,

Agreed. either x % 2^N or x & (2^N - 1) is fine, but we use the wrong x % (2^N - 1) here.

Quoted 9 lines
Quoted 1 line
Quoted 1 line

The expression full = (cons == ((qp->cur_index + 1) % q->index_mask)) is correct.

Actually you are correct. If you look at queue_full() it uses full = (cons == (prod + 1)& q->index_mask) and queue_empty() uses empty = (prod == cons) *but*
check_qp_queue_full() uses
full = (cons == ((qp->cur_index + 1) % q->index_mask)) with % instead of&. I was so used to looking at the first two I missed the error in the last one.

% should be& like the others.

I will update the commit message and send v2 patch.

Thanks lot.

Best Regards,
Xiao Yang

Quoted 3 lines

Bob