Skip to contentSkip to conversation

RXE status in the upstream rping using rxe

DISCUSSION Leon Romanovsky 5 participants 19 messages Aug 3, 2021 – Aug 23, 2021 InfiniBand

lore
Share & tools
lore
#0 lore raw

Hi,

Can you please help me to understand the RXE status in the upstream?

Does we still have crashes/interop issues/e.t.c?

Latest commit is:
20da44dfe8ef ("RDMA/mlx5: Drop in-driver verbs object creations")

Thanks

#1 lore raw
Quoted 6 lines · Leon Romanovsky

Hi,

Can you please help me to understand the RXE status in the upstream?

Does we still have crashes/interop issues/e.t.c?

I made some developments with the RXE in the upstream, from my usage with latest RXE,
I found the following:

1. rdma-core can not work well with latest RDMA git;
2. There are some problems with rxe in different kernel version;

To 2, the commit
https://patchwork.kernel.org/project/linux-rdma/patch/[email protected]/
can relieve this problem,
not sure if some bugs still exist in RXE.

To 1, I will continue to make further investigations.
So we should have time to make tests, fix bugs and make rxe stable.

Zhu Yanjun

Quoted 5 lines

Latest commit is:
20da44dfe8ef ("RDMA/mlx5: Drop in-driver verbs object creations")

Thanks

#2 lore raw
Quoted 7 lines · Zhu Yanjun
Quoted 5 lines · Leon Romanovsky

Hi,

Can you please help me to understand the RXE status in the upstream?

Quoted 7 lines
Quoted 1 line

Does we still have crashes/interop issues/e.t.c?

I made some developments with the RXE in the upstream, from my usage with latest RXE,
I found the following:

1. rdma-core can not work well with latest RDMA git;

The latest RDMA git is
https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git

Zhu Yanjun

Quoted 16 lines

2. There are some problems with rxe in different kernel version;

To 2, the commit
https://patchwork.kernel.org/project/linux-rdma/patch/[email protected]/
can relieve this problem,
not sure if some bugs still exist in RXE.

To 1, I will continue to make further investigations.
So we should have time to make tests, fix bugs and make rxe stable.

Zhu Yanjun

Quoted 5 lines

Latest commit is:
20da44dfe8ef ("RDMA/mlx5: Drop in-driver verbs object creations")

Thanks

#3 lore raw
Quoted 10 lines · Zhu Yanjun
Quoted 9 lines · Zhu Yanjun
Quoted 6 lines · Leon Romanovsky

Hi,

Can you please help me to understand the RXE status in the upstream?

Does we still have crashes/interop issues/e.t.c?

Quoted 8 lines
Quoted 5 lines

I made some developments with the RXE in the upstream, from my usage with latest RXE,
I found the following:

1. rdma-core can not work well with latest RDMA git;

The latest RDMA git is
https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git

"Latest" is a relative term, what SHA did you test?
Let's focus on fixing RXE before we will continue with new features.

Thanks

#4 lore raw
Quoted 16 lines · Leon Romanovsky
Quoted 14 lines · Zhu Yanjun
Quoted 13 lines · Zhu Yanjun
Quoted 6 lines · Leon Romanovsky

Hi,

Can you please help me to understand the RXE status in the upstream?

Does we still have crashes/interop issues/e.t.c?

I made some developments with the RXE in the upstream, from my usage with latest RXE,
I found the following:

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

1. rdma-core can not work well with latest RDMA git;

The latest RDMA git is
https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git

"Latest" is a relative term, what SHA did you test?
Let's focus on fixing RXE before we will continue with new features.

Thanks a lot. I agree with you.

rdma-core:
313509f8 (HEAD -> master, origin/master, origin/HEAD) Merge pull request #1038 from selvintxavier/master
2d3dc48b Merge pull request #1039 from amzn/pyverbs-mac-fix-pr 327d45e0 tests: Add missing MAC element to args list 66aba73d bnxt_re/lib: Move hardware queue to 16B aligned indices 8754fb51 bnxt_re/lib: Use separate indices for shadow queue be4d8abf bnxt_re/lib: add a function to initialize software queue

kernel rdma:
0050a57638ca (HEAD -> for-next, origin/for-next, origin/HEAD) RDMA/qedr: Improve error logs for rdma_alloc_tid error return 090473004b02 RDMA/qed: Use accurate error num in qed_cxt_dynamic_ilt_alloc 991c4274dc17 RDMA/hfi1: Fix typo in comments
8d7e415d5561 docs: Fix infiniband uverbs minor number bbafcbc2b1c9 RDMA/iwpm: Rely on the rdma_nl_[un]register() to ensure that requests are valid
bdb0e4e3ff19 RDMA/iwpm: Remove not-needed reference counting e677b72a0647 RDMA/iwcm: Release resources if iw_cm module initialization fails a0293eb24936 RDMA/hfi1: Convert from atomic_t to refcount_t on hfi1_devdata->user_refcount

with the above kernel and rdma-core, the following messages will appear. "

[   54.214608] rdma_rxe: loaded
[   54.217089] infiniband rxe0: set active
[   54.217101] infiniband rxe0: added enp0s8
[  167.623200] rdma_rxe: cqe(32768) > max_cqe(32767)
[  167.645590] rdma_rxe: cqe(1) < current # elements in queue (6)
[  167.733297] rdma_rxe: cqe(32768) > max_cqe(32767)
[  169.074755] rdma_rxe: check_rkey: no MW matches rkey 0x1000247
[  169.074796] rdma_rxe: qp#27 moved to error state
[  169.138851] rdma_rxe: check_rkey: no MW matches rkey 0x10005de
[  169.138889] rdma_rxe: qp#30 moved to error state
[  169.160565] rdma_rxe: check_rkey: no MW matches rkey 0x10006f7
[  169.160601] rdma_rxe: qp#31 moved to error state
[  169.182132] rdma_rxe: check_rkey: no MW matches rkey 0x1000782
[  169.182170] rdma_rxe: qp#32 moved to error state
[  169.667803] rdma_rxe: check_rkey: no MR matches rkey 0x18d8
[  169.667850] rdma_rxe: qp#39 moved to error state
[  198.872649] rdma_rxe: cqe(32768) > max_cqe(32767)
[  198.894829] rdma_rxe: cqe(1) < current # elements in queue (6)
[  198.981839] rdma_rxe: cqe(32768) > max_cqe(32767)
[  200.332031] rdma_rxe: check_rkey: no MW matches rkey 0x1000887
[  200.332086] rdma_rxe: qp#58 moved to error state
[  200.396476] rdma_rxe: check_rkey: no MW matches rkey 0x1000b0d
[  200.396514] rdma_rxe: qp#61 moved to error state
[  200.417919] rdma_rxe: check_rkey: no MW matches rkey 0x1000c40
[  200.417956] rdma_rxe: qp#62 moved to error state
[  200.439616] rdma_rxe: check_rkey: no MW matches rkey 0x1000d24
[  200.439654] rdma_rxe: qp#63 moved to error state
[  200.933104] rdma_rxe: check_rkey: no MR matches rkey 0x37d8
[  200.933153] rdma_rxe: qp#70 moved to error state
[  206.880305] rdma_rxe: cqe(32768) > max_cqe(32767)
[  206.904030] rdma_rxe: cqe(1) < current # elements in queue (6)
[  206.991494] rdma_rxe: cqe(32768) > max_cqe(32767)
[  208.359987] rdma_rxe: check_rkey: no MW matches rkey 0x1000e4d
[  208.360028] rdma_rxe: qp#89 moved to error state
[  208.425637] rdma_rxe: check_rkey: no MW matches rkey 0x1001136
[  208.425675] rdma_rxe: qp#92 moved to error state
[  208.447333] rdma_rxe: check_rkey: no MW matches rkey 0x10012d8
[  208.447370] rdma_rxe: qp#93 moved to error state
[  208.469511] rdma_rxe: check_rkey: no MW matches rkey 0x100137a
[  208.469550] rdma_rxe: qp#94 moved to error state
[  208.956691] rdma_rxe: check_rkey: no MR matches rkey 0x5670
[  208.956731] rdma_rxe: qp#100 moved to error state
[  216.879703] rdma_rxe: cqe(32768) > max_cqe(32767)
[  216.902199] rdma_rxe: cqe(1) < current # elements in queue (6)
[  216.989264] rdma_rxe: cqe(32768) > max_cqe(32767)
[  218.363765] rdma_rxe: check_rkey: no MW matches rkey 0x10014d6
[  218.363808] rdma_rxe: qp#119 moved to error state
[  218.429474] rdma_rxe: check_rkey: no MW matches rkey 0x10017e4
[  218.429513] rdma_rxe: qp#122 moved to error state
[  218.451443] rdma_rxe: check_rkey: no MW matches rkey 0x1001895
[  218.451481] rdma_rxe: qp#123 moved to error state
[  218.473869] rdma_rxe: check_rkey: no MW matches rkey 0x1001910
[  218.473908] rdma_rxe: qp#124 moved to error state
[  218.963602] rdma_rxe: check_rkey: no MR matches rkey 0x757b
[  218.963641] rdma_rxe: qp#130 moved to error state
[  233.855140] rdma_rxe: cqe(32768) > max_cqe(32767)
[  233.877202] rdma_rxe: cqe(1) < current # elements in queue (6)
[  233.963952] rdma_rxe: cqe(32768) > max_cqe(32767)
[  235.305274] rdma_rxe: check_rkey: no MW matches rkey 0x1001ac2
[  235.305319] rdma_rxe: qp#149 moved to error state
[  235.368800] rdma_rxe: check_rkey: no MW matches rkey 0x1001db8
[  235.368838] rdma_rxe: qp#152 moved to error state
[  235.390155] rdma_rxe: check_rkey: no MW matches rkey 0x1001e4d
[  235.390192] rdma_rxe: qp#153 moved to error state
[  235.411336] rdma_rxe: check_rkey: no MW matches rkey 0x1001f4c
[  235.411374] rdma_rxe: qp#154 moved to error state
[  235.895784] rdma_rxe: check_rkey: no MR matches rkey 0x9482
[  235.895828] rdma_rxe: qp#161 moved to error state

"
Not sure if they are problems.
IMO, we should make further investigations.

Thanks
Zhu Yanjun

Quoted 2 lines

Thanks

#5 lore raw
Quoted 20 lines · Zhu Yanjun
Quoted 18 lines · Leon Romanovsky
Quoted 16 lines · Zhu Yanjun
Quoted 14 lines · Zhu Yanjun
Quoted 6 lines · Leon Romanovsky

Hi,

Can you please help me to understand the RXE status in the upstream?

Does we still have crashes/interop issues/e.t.c?

I made some developments with the RXE in the upstream, from my usage with latest RXE,
I found the following:

1. rdma-core can not work well with latest RDMA git;

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

The latest RDMA git is
https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git

"Latest" is a relative term, what SHA did you test?
Let's focus on fixing RXE before we will continue with new features.

Thanks a lot. I agree with you.

I believe simple rping still doesn't work linux-to-linux. The last working version (of rping in rxe) was 5.13 I think. I have posted a number of crashes rping encounters (gotta get that working before I can even try NFSoRDMA).

Thank you for working on the code.

We (NFS community) do test NFSoRDMA every git pull using rxe and siw but lately have been encountering problems.

Quoted 100 lines

rdma-core:
313509f8 (HEAD -> master, origin/master, origin/HEAD) Merge pull request #1038 from selvintxavier/master
2d3dc48b Merge pull request #1039 from amzn/pyverbs-mac-fix-pr 327d45e0 tests: Add missing MAC element to args list 66aba73d bnxt_re/lib: Move hardware queue to 16B aligned indices 8754fb51 bnxt_re/lib: Use separate indices for shadow queue be4d8abf bnxt_re/lib: add a function to initialize software queue

kernel rdma:
0050a57638ca (HEAD -> for-next, origin/for-next, origin/HEAD) RDMA/qedr: Improve error logs for rdma_alloc_tid error return 090473004b02 RDMA/qed: Use accurate error num in qed_cxt_dynamic_ilt_alloc 991c4274dc17 RDMA/hfi1: Fix typo in comments
8d7e415d5561 docs: Fix infiniband uverbs minor number bbafcbc2b1c9 RDMA/iwpm: Rely on the rdma_nl_[un]register() to ensure that requests are valid
bdb0e4e3ff19 RDMA/iwpm: Remove not-needed reference counting e677b72a0647 RDMA/iwcm: Release resources if iw_cm module initialization fails a0293eb24936 RDMA/hfi1: Convert from atomic_t to refcount_t on hfi1_devdata->user_refcount

with the above kernel and rdma-core, the following messages will appear. "

[   54.214608] rdma_rxe: loaded
[   54.217089] infiniband rxe0: set active
[   54.217101] infiniband rxe0: added enp0s8
[  167.623200] rdma_rxe: cqe(32768) > max_cqe(32767)
[  167.645590] rdma_rxe: cqe(1) < current # elements in queue (6)
[  167.733297] rdma_rxe: cqe(32768) > max_cqe(32767)
[  169.074755] rdma_rxe: check_rkey: no MW matches rkey 0x1000247
[  169.074796] rdma_rxe: qp#27 moved to error state
[  169.138851] rdma_rxe: check_rkey: no MW matches rkey 0x10005de
[  169.138889] rdma_rxe: qp#30 moved to error state
[  169.160565] rdma_rxe: check_rkey: no MW matches rkey 0x10006f7
[  169.160601] rdma_rxe: qp#31 moved to error state
[  169.182132] rdma_rxe: check_rkey: no MW matches rkey 0x1000782
[  169.182170] rdma_rxe: qp#32 moved to error state
[  169.667803] rdma_rxe: check_rkey: no MR matches rkey 0x18d8
[  169.667850] rdma_rxe: qp#39 moved to error state
[  198.872649] rdma_rxe: cqe(32768) > max_cqe(32767)
[  198.894829] rdma_rxe: cqe(1) < current # elements in queue (6)
[  198.981839] rdma_rxe: cqe(32768) > max_cqe(32767)
[  200.332031] rdma_rxe: check_rkey: no MW matches rkey 0x1000887
[  200.332086] rdma_rxe: qp#58 moved to error state
[  200.396476] rdma_rxe: check_rkey: no MW matches rkey 0x1000b0d
[  200.396514] rdma_rxe: qp#61 moved to error state
[  200.417919] rdma_rxe: check_rkey: no MW matches rkey 0x1000c40
[  200.417956] rdma_rxe: qp#62 moved to error state
[  200.439616] rdma_rxe: check_rkey: no MW matches rkey 0x1000d24
[  200.439654] rdma_rxe: qp#63 moved to error state
[  200.933104] rdma_rxe: check_rkey: no MR matches rkey 0x37d8
[  200.933153] rdma_rxe: qp#70 moved to error state
[  206.880305] rdma_rxe: cqe(32768) > max_cqe(32767)
[  206.904030] rdma_rxe: cqe(1) < current # elements in queue (6)
[  206.991494] rdma_rxe: cqe(32768) > max_cqe(32767)
[  208.359987] rdma_rxe: check_rkey: no MW matches rkey 0x1000e4d
[  208.360028] rdma_rxe: qp#89 moved to error state
[  208.425637] rdma_rxe: check_rkey: no MW matches rkey 0x1001136
[  208.425675] rdma_rxe: qp#92 moved to error state
[  208.447333] rdma_rxe: check_rkey: no MW matches rkey 0x10012d8
[  208.447370] rdma_rxe: qp#93 moved to error state
[  208.469511] rdma_rxe: check_rkey: no MW matches rkey 0x100137a
[  208.469550] rdma_rxe: qp#94 moved to error state
[  208.956691] rdma_rxe: check_rkey: no MR matches rkey 0x5670
[  208.956731] rdma_rxe: qp#100 moved to error state
[  216.879703] rdma_rxe: cqe(32768) > max_cqe(32767)
[  216.902199] rdma_rxe: cqe(1) < current # elements in queue (6)
[  216.989264] rdma_rxe: cqe(32768) > max_cqe(32767)
[  218.363765] rdma_rxe: check_rkey: no MW matches rkey 0x10014d6
[  218.363808] rdma_rxe: qp#119 moved to error state
[  218.429474] rdma_rxe: check_rkey: no MW matches rkey 0x10017e4
[  218.429513] rdma_rxe: qp#122 moved to error state
[  218.451443] rdma_rxe: check_rkey: no MW matches rkey 0x1001895
[  218.451481] rdma_rxe: qp#123 moved to error state
[  218.473869] rdma_rxe: check_rkey: no MW matches rkey 0x1001910
[  218.473908] rdma_rxe: qp#124 moved to error state
[  218.963602] rdma_rxe: check_rkey: no MR matches rkey 0x757b
[  218.963641] rdma_rxe: qp#130 moved to error state
[  233.855140] rdma_rxe: cqe(32768) > max_cqe(32767)
[  233.877202] rdma_rxe: cqe(1) < current # elements in queue (6)
[  233.963952] rdma_rxe: cqe(32768) > max_cqe(32767)
[  235.305274] rdma_rxe: check_rkey: no MW matches rkey 0x1001ac2
[  235.305319] rdma_rxe: qp#149 moved to error state
[  235.368800] rdma_rxe: check_rkey: no MW matches rkey 0x1001db8
[  235.368838] rdma_rxe: qp#152 moved to error state
[  235.390155] rdma_rxe: check_rkey: no MW matches rkey 0x1001e4d
[  235.390192] rdma_rxe: qp#153 moved to error state
[  235.411336] rdma_rxe: check_rkey: no MW matches rkey 0x1001f4c
[  235.411374] rdma_rxe: qp#154 moved to error state
[  235.895784] rdma_rxe: check_rkey: no MR matches rkey 0x9482
[  235.895828] rdma_rxe: qp#161 moved to error state

"
Not sure if they are problems.
IMO, we should make further investigations.

Thanks
Zhu Yanjun

Quoted 2 lines

Thanks

#16 lore raw
In reply to Zhu Yanjun
Quoted 119 lines · Zhu Yanjun
Quoted 16 lines · Leon Romanovsky
Quoted 14 lines · Zhu Yanjun
Quoted 13 lines · Zhu Yanjun
Quoted 6 lines · Leon Romanovsky

Hi,

Can you please help me to understand the RXE status in the upstream?

Does we still have crashes/interop issues/e.t.c?

I made some developments with the RXE in the upstream, from my usage with latest RXE,
I found the following:

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

1. rdma-core can not work well with latest RDMA git;

The latest RDMA git is
https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git

"Latest" is a relative term, what SHA did you test?
Let's focus on fixing RXE before we will continue with new features.

Thanks a lot. I agree with you.

rdma-core:
313509f8 (HEAD -> master, origin/master, origin/HEAD) Merge pull request #1038 from selvintxavier/master
2d3dc48b Merge pull request #1039 from amzn/pyverbs-mac-fix-pr 327d45e0 tests: Add missing MAC element to args list 66aba73d bnxt_re/lib: Move hardware queue to 16B aligned indices 8754fb51 bnxt_re/lib: Use separate indices for shadow queue be4d8abf bnxt_re/lib: add a function to initialize software queue

kernel rdma:
0050a57638ca (HEAD -> for-next, origin/for-next, origin/HEAD) RDMA/qedr: Improve error logs for rdma_alloc_tid error return 090473004b02 RDMA/qed: Use accurate error num in qed_cxt_dynamic_ilt_alloc 991c4274dc17 RDMA/hfi1: Fix typo in comments
8d7e415d5561 docs: Fix infiniband uverbs minor number bbafcbc2b1c9 RDMA/iwpm: Rely on the rdma_nl_[un]register() to ensure that requests are valid
bdb0e4e3ff19 RDMA/iwpm: Remove not-needed reference counting e677b72a0647 RDMA/iwcm: Release resources if iw_cm module initialization fails a0293eb24936 RDMA/hfi1: Convert from atomic_t to refcount_t on hfi1_devdata->user_refcount

with the above kernel and rdma-core, the following messages will appear. "

[   54.214608] rdma_rxe: loaded
[   54.217089] infiniband rxe0: set active
[   54.217101] infiniband rxe0: added enp0s8
[  167.623200] rdma_rxe: cqe(32768) > max_cqe(32767)
[  167.645590] rdma_rxe: cqe(1) < current # elements in queue (6)
[  167.733297] rdma_rxe: cqe(32768) > max_cqe(32767)
[  169.074755] rdma_rxe: check_rkey: no MW matches rkey 0x1000247
[  169.074796] rdma_rxe: qp#27 moved to error state
[  169.138851] rdma_rxe: check_rkey: no MW matches rkey 0x10005de
[  169.138889] rdma_rxe: qp#30 moved to error state
[  169.160565] rdma_rxe: check_rkey: no MW matches rkey 0x10006f7
[  169.160601] rdma_rxe: qp#31 moved to error state
[  169.182132] rdma_rxe: check_rkey: no MW matches rkey 0x1000782
[  169.182170] rdma_rxe: qp#32 moved to error state
[  169.667803] rdma_rxe: check_rkey: no MR matches rkey 0x18d8
[  169.667850] rdma_rxe: qp#39 moved to error state
[  198.872649] rdma_rxe: cqe(32768) > max_cqe(32767)
[  198.894829] rdma_rxe: cqe(1) < current # elements in queue (6)
[  198.981839] rdma_rxe: cqe(32768) > max_cqe(32767)
[  200.332031] rdma_rxe: check_rkey: no MW matches rkey 0x1000887
[  200.332086] rdma_rxe: qp#58 moved to error state
[  200.396476] rdma_rxe: check_rkey: no MW matches rkey 0x1000b0d
[  200.396514] rdma_rxe: qp#61 moved to error state
[  200.417919] rdma_rxe: check_rkey: no MW matches rkey 0x1000c40
[  200.417956] rdma_rxe: qp#62 moved to error state
[  200.439616] rdma_rxe: check_rkey: no MW matches rkey 0x1000d24
[  200.439654] rdma_rxe: qp#63 moved to error state
[  200.933104] rdma_rxe: check_rkey: no MR matches rkey 0x37d8
[  200.933153] rdma_rxe: qp#70 moved to error state
[  206.880305] rdma_rxe: cqe(32768) > max_cqe(32767)
[  206.904030] rdma_rxe: cqe(1) < current # elements in queue (6)
[  206.991494] rdma_rxe: cqe(32768) > max_cqe(32767)
[  208.359987] rdma_rxe: check_rkey: no MW matches rkey 0x1000e4d
[  208.360028] rdma_rxe: qp#89 moved to error state
[  208.425637] rdma_rxe: check_rkey: no MW matches rkey 0x1001136
[  208.425675] rdma_rxe: qp#92 moved to error state
[  208.447333] rdma_rxe: check_rkey: no MW matches rkey 0x10012d8
[  208.447370] rdma_rxe: qp#93 moved to error state
[  208.469511] rdma_rxe: check_rkey: no MW matches rkey 0x100137a
[  208.469550] rdma_rxe: qp#94 moved to error state
[  208.956691] rdma_rxe: check_rkey: no MR matches rkey 0x5670
[  208.956731] rdma_rxe: qp#100 moved to error state
[  216.879703] rdma_rxe: cqe(32768) > max_cqe(32767)
[  216.902199] rdma_rxe: cqe(1) < current # elements in queue (6)
[  216.989264] rdma_rxe: cqe(32768) > max_cqe(32767)
[  218.363765] rdma_rxe: check_rkey: no MW matches rkey 0x10014d6
[  218.363808] rdma_rxe: qp#119 moved to error state
[  218.429474] rdma_rxe: check_rkey: no MW matches rkey 0x10017e4
[  218.429513] rdma_rxe: qp#122 moved to error state
[  218.451443] rdma_rxe: check_rkey: no MW matches rkey 0x1001895
[  218.451481] rdma_rxe: qp#123 moved to error state
[  218.473869] rdma_rxe: check_rkey: no MW matches rkey 0x1001910
[  218.473908] rdma_rxe: qp#124 moved to error state
[  218.963602] rdma_rxe: check_rkey: no MR matches rkey 0x757b
[  218.963641] rdma_rxe: qp#130 moved to error state
[  233.855140] rdma_rxe: cqe(32768) > max_cqe(32767)
[  233.877202] rdma_rxe: cqe(1) < current # elements in queue (6)
[  233.963952] rdma_rxe: cqe(32768) > max_cqe(32767)
[  235.305274] rdma_rxe: check_rkey: no MW matches rkey 0x1001ac2
[  235.305319] rdma_rxe: qp#149 moved to error state
[  235.368800] rdma_rxe: check_rkey: no MW matches rkey 0x1001db8
[  235.368838] rdma_rxe: qp#152 moved to error state
[  235.390155] rdma_rxe: check_rkey: no MW matches rkey 0x1001e4d
[  235.390192] rdma_rxe: qp#153 moved to error state
[  235.411336] rdma_rxe: check_rkey: no MW matches rkey 0x1001f4c
[  235.411374] rdma_rxe: qp#154 moved to error state
[  235.895784] rdma_rxe: check_rkey: no MR matches rkey 0x9482
[  235.895828] rdma_rxe: qp#161 moved to error state
Quoted 8 lines

"
Not sure if they are problems.
IMO, we should make further investigations.

Thanks
Zhu Yanjun

Quoted 2 lines

Thanks

All of the messages are from the rxe driver caused by the python tests intentionally causing

errors. Here is a test run with messages. No errors occurred. This is run on current rdma_core and for_next. Does not answer the question about rping. That needs more testing.
(so ru is short for "./build/bin/run_tests.py --dev rxe_1")

Bob

rpearson:rdma-core$ sudo dmesg -C

rpearson:rdma-core$ so ru

.............sssssssss.............sssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........sssssssssssssssssss....ssss........s...s.s..s..........ssssssssss..ss

----------------------------------------------------------------------

Ran 199 tests in 0.418s

OK (skipped=134)

rpearson:rdma-core$ sudo dmesg

[ 9396.038090] rdma_rxe: cqe(32768) > max_cqe(32767)
[ 9396.042414] rdma_rxe: cqe(1) < current # elements in queue (6)
[ 9396.056685] rdma_rxe: cqe(32768) > max_cqe(32767)
[ 9396.273114] rdma_rxe: check_rkey: no MW matches rkey 0x1000256
[ 9396.273120] rdma_rxe: qp#27 moved to error state
[ 9396.283112] rdma_rxe: check_rkey: no MW matches rkey 0x10005be
[ 9396.283116] rdma_rxe: qp#30 moved to error state
[ 9396.286497] rdma_rxe: check_rkey: no MW matches rkey 0x100063d
[ 9396.286501] rdma_rxe: qp#31 moved to error state
[ 9396.289917] rdma_rxe: check_rkey: no MW matches rkey 0x10007a6
[ 9396.289922] rdma_rxe: qp#32 moved to error state
[ 9396.364850] rdma_rxe: check_rkey: no MR matches rkey 0x1868
[ 9396.364854] rdma_rxe: qp#37 moved to error state
#17 lore raw
Quoted 173 lines · Bob Pearson
Quoted 119 lines · Zhu Yanjun
Quoted 16 lines · Leon Romanovsky
Quoted 14 lines · Zhu Yanjun
Quoted 13 lines · Zhu Yanjun
Quoted 6 lines · Leon Romanovsky

Hi,

Can you please help me to understand the RXE status in the upstream?

Does we still have crashes/interop issues/e.t.c?

I made some developments with the RXE in the upstream, from my usage with latest RXE,
I found the following:

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

1. rdma-core can not work well with latest RDMA git;

The latest RDMA git is
https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git

"Latest" is a relative term, what SHA did you test?
Let's focus on fixing RXE before we will continue with new features.

Thanks a lot. I agree with you.

rdma-core:
313509f8 (HEAD -> master, origin/master, origin/HEAD) Merge pull request #1038 from selvintxavier/master
2d3dc48b Merge pull request #1039 from amzn/pyverbs-mac-fix-pr 327d45e0 tests: Add missing MAC element to args list 66aba73d bnxt_re/lib: Move hardware queue to 16B aligned indices 8754fb51 bnxt_re/lib: Use separate indices for shadow queue be4d8abf bnxt_re/lib: add a function to initialize software queue

kernel rdma:
0050a57638ca (HEAD -> for-next, origin/for-next, origin/HEAD) RDMA/qedr: Improve error logs for rdma_alloc_tid error return 090473004b02 RDMA/qed: Use accurate error num in qed_cxt_dynamic_ilt_alloc 991c4274dc17 RDMA/hfi1: Fix typo in comments
8d7e415d5561 docs: Fix infiniband uverbs minor number bbafcbc2b1c9 RDMA/iwpm: Rely on the rdma_nl_[un]register() to ensure that requests are valid
bdb0e4e3ff19 RDMA/iwpm: Remove not-needed reference counting e677b72a0647 RDMA/iwcm: Release resources if iw_cm module initialization fails a0293eb24936 RDMA/hfi1: Convert from atomic_t to refcount_t on hfi1_devdata->user_refcount

with the above kernel and rdma-core, the following messages will appear. "

[   54.214608] rdma_rxe: loaded
[   54.217089] infiniband rxe0: set active
[   54.217101] infiniband rxe0: added enp0s8
[  167.623200] rdma_rxe: cqe(32768) > max_cqe(32767)
[  167.645590] rdma_rxe: cqe(1) < current # elements in queue (6)
[  167.733297] rdma_rxe: cqe(32768) > max_cqe(32767)
[  169.074755] rdma_rxe: check_rkey: no MW matches rkey 0x1000247
[  169.074796] rdma_rxe: qp#27 moved to error state
[  169.138851] rdma_rxe: check_rkey: no MW matches rkey 0x10005de
[  169.138889] rdma_rxe: qp#30 moved to error state
[  169.160565] rdma_rxe: check_rkey: no MW matches rkey 0x10006f7
[  169.160601] rdma_rxe: qp#31 moved to error state
[  169.182132] rdma_rxe: check_rkey: no MW matches rkey 0x1000782
[  169.182170] rdma_rxe: qp#32 moved to error state
[  169.667803] rdma_rxe: check_rkey: no MR matches rkey 0x18d8
[  169.667850] rdma_rxe: qp#39 moved to error state
[  198.872649] rdma_rxe: cqe(32768) > max_cqe(32767)
[  198.894829] rdma_rxe: cqe(1) < current # elements in queue (6)
[  198.981839] rdma_rxe: cqe(32768) > max_cqe(32767)
[  200.332031] rdma_rxe: check_rkey: no MW matches rkey 0x1000887
[  200.332086] rdma_rxe: qp#58 moved to error state
[  200.396476] rdma_rxe: check_rkey: no MW matches rkey 0x1000b0d
[  200.396514] rdma_rxe: qp#61 moved to error state
[  200.417919] rdma_rxe: check_rkey: no MW matches rkey 0x1000c40
[  200.417956] rdma_rxe: qp#62 moved to error state
[  200.439616] rdma_rxe: check_rkey: no MW matches rkey 0x1000d24
[  200.439654] rdma_rxe: qp#63 moved to error state
[  200.933104] rdma_rxe: check_rkey: no MR matches rkey 0x37d8
[  200.933153] rdma_rxe: qp#70 moved to error state
[  206.880305] rdma_rxe: cqe(32768) > max_cqe(32767)
[  206.904030] rdma_rxe: cqe(1) < current # elements in queue (6)
[  206.991494] rdma_rxe: cqe(32768) > max_cqe(32767)
[  208.359987] rdma_rxe: check_rkey: no MW matches rkey 0x1000e4d
[  208.360028] rdma_rxe: qp#89 moved to error state
[  208.425637] rdma_rxe: check_rkey: no MW matches rkey 0x1001136
[  208.425675] rdma_rxe: qp#92 moved to error state
[  208.447333] rdma_rxe: check_rkey: no MW matches rkey 0x10012d8
[  208.447370] rdma_rxe: qp#93 moved to error state
[  208.469511] rdma_rxe: check_rkey: no MW matches rkey 0x100137a
[  208.469550] rdma_rxe: qp#94 moved to error state
[  208.956691] rdma_rxe: check_rkey: no MR matches rkey 0x5670
[  208.956731] rdma_rxe: qp#100 moved to error state
[  216.879703] rdma_rxe: cqe(32768) > max_cqe(32767)
[  216.902199] rdma_rxe: cqe(1) < current # elements in queue (6)
[  216.989264] rdma_rxe: cqe(32768) > max_cqe(32767)
[  218.363765] rdma_rxe: check_rkey: no MW matches rkey 0x10014d6
[  218.363808] rdma_rxe: qp#119 moved to error state
[  218.429474] rdma_rxe: check_rkey: no MW matches rkey 0x10017e4
[  218.429513] rdma_rxe: qp#122 moved to error state
[  218.451443] rdma_rxe: check_rkey: no MW matches rkey 0x1001895
[  218.451481] rdma_rxe: qp#123 moved to error state
[  218.473869] rdma_rxe: check_rkey: no MW matches rkey 0x1001910
[  218.473908] rdma_rxe: qp#124 moved to error state
[  218.963602] rdma_rxe: check_rkey: no MR matches rkey 0x757b
[  218.963641] rdma_rxe: qp#130 moved to error state
[  233.855140] rdma_rxe: cqe(32768) > max_cqe(32767)
[  233.877202] rdma_rxe: cqe(1) < current # elements in queue (6)
[  233.963952] rdma_rxe: cqe(32768) > max_cqe(32767)
[  235.305274] rdma_rxe: check_rkey: no MW matches rkey 0x1001ac2
[  235.305319] rdma_rxe: qp#149 moved to error state
[  235.368800] rdma_rxe: check_rkey: no MW matches rkey 0x1001db8
[  235.368838] rdma_rxe: qp#152 moved to error state
[  235.390155] rdma_rxe: check_rkey: no MW matches rkey 0x1001e4d
[  235.390192] rdma_rxe: qp#153 moved to error state
[  235.411336] rdma_rxe: check_rkey: no MW matches rkey 0x1001f4c
[  235.411374] rdma_rxe: qp#154 moved to error state
[  235.895784] rdma_rxe: check_rkey: no MR matches rkey 0x9482
[  235.895828] rdma_rxe: qp#161 moved to error state
Quoted 8 lines

"
Not sure if they are problems.
IMO, we should make further investigations.

Thanks
Zhu Yanjun

Quoted 2 lines

Thanks

All of the messages are from the rxe driver caused by the python tests intentionally causing

errors. Here is a test run with messages. No errors occurred. This is run on current rdma_core and for_next. Does not answer the question about rping. That needs more testing.
(so ru is short for "./build/bin/run_tests.py --dev rxe_1")

Bob

rpearson:rdma-core$ sudo dmesg -C

rpearson:rdma-core$ so ru

.............sssssssss.............sssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss........sssssssssssssssssss....ssss........s...s.s..s..........ssssssssss..ss

----------------------------------------------------------------------

Ran 199 tests in 0.418s

OK (skipped=134)

rpearson:rdma-core$ sudo dmesg

[ 9396.038090] rdma_rxe: cqe(32768) > max_cqe(32767)
[ 9396.042414] rdma_rxe: cqe(1) < current # elements in queue (6)
[ 9396.056685] rdma_rxe: cqe(32768) > max_cqe(32767)
[ 9396.273114] rdma_rxe: check_rkey: no MW matches rkey 0x1000256
[ 9396.273120] rdma_rxe: qp#27 moved to error state
[ 9396.283112] rdma_rxe: check_rkey: no MW matches rkey 0x10005be
[ 9396.283116] rdma_rxe: qp#30 moved to error state
[ 9396.286497] rdma_rxe: check_rkey: no MW matches rkey 0x100063d
[ 9396.286501] rdma_rxe: qp#31 moved to error state
Quoted 7 lines
[ 9396.289917] rdma_rxe: check_rkey: no MW matches rkey 0x10007a6
[ 9396.289922] rdma_rxe: qp#32 moved to error state
[ 9396.364850] rdma_rxe: check_rkey: no MR matches rkey 0x1868
[ 9396.364854] rdma_rxe: qp#37 moved to error state

You shouldn't print these errors by default, they need be *_dbg() level,

Thanks

Quoted 4 lines
#6 lore raw
In reply to Olga Kornievskaia
Quoted 28 lines · Olga Kornievskaia
Quoted 26 lines · Zhu Yanjun
Quoted 23 lines · Leon Romanovsky
Quoted 10 lines · Zhu Yanjun
Quoted 9 lines · Zhu Yanjun
Quoted 6 lines · Leon Romanovsky

Hi,

Can you please help me to understand the RXE status in the upstream?

Does we still have crashes/interop issues/e.t.c?

Quoted 8 lines
Quoted 5 lines

I made some developments with the RXE in the upstream, from my usage with latest RXE,
I found the following:

1. rdma-core can not work well with latest RDMA git;

The latest RDMA git is
https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git

"Latest" is a relative term, what SHA did you test?
Let's focus on fixing RXE before we will continue with new features.

Quoted 6 lines
Quoted 1 line

Thanks a lot. I agree with you.

I believe simple rping still doesn't work linux-to-linux. The last working version (of rping in rxe) was 5.13 I think. I have posted a number of crashes rping encounters (gotta get that working before I can even try NFSoRDMA).

The following are my tests.

1. Modprobe rdma_rxe
2. Modprobe -v -r rdma_rxe
3. Rdma link add rxe
4. Rdma link del rxe
5. Latest rdma-core && latest kernel upstream;
6. Latest kernel < ------rping---- > 5.10.y stable
7. Latest kernel < ------rping---- > 5.11.y stable
8. Latest kernel < ------rping---- > 5.12.y stable
9. Latest kernel < ------rping---- > 5.13.y stable

It seems that the latest kernel upstream (5.14-rc6) can rping other stable kernels.
Can you make tests again?

Zhu Yanjun

Quoted 106 lines

Thank you for working on the code.

We (NFS community) do test NFSoRDMA every git pull using rxe and siw but lately have been encountering problems.

Quoted 100 lines

rdma-core:
313509f8 (HEAD -> master, origin/master, origin/HEAD) Merge pull request #1038 from selvintxavier/master
2d3dc48b Merge pull request #1039 from amzn/pyverbs-mac-fix-pr 327d45e0 tests: Add missing MAC element to args list 66aba73d bnxt_re/lib: Move hardware queue to 16B aligned indices 8754fb51 bnxt_re/lib: Use separate indices for shadow queue be4d8abf bnxt_re/lib: add a function to initialize software queue

kernel rdma:
0050a57638ca (HEAD -> for-next, origin/for-next, origin/HEAD) RDMA/qedr: Improve error logs for rdma_alloc_tid error return 090473004b02 RDMA/qed: Use accurate error num in qed_cxt_dynamic_ilt_alloc 991c4274dc17 RDMA/hfi1: Fix typo in comments
8d7e415d5561 docs: Fix infiniband uverbs minor number bbafcbc2b1c9 RDMA/iwpm: Rely on the rdma_nl_[un]register() to ensure that requests are valid
bdb0e4e3ff19 RDMA/iwpm: Remove not-needed reference counting e677b72a0647 RDMA/iwcm: Release resources if iw_cm module initialization fails a0293eb24936 RDMA/hfi1: Convert from atomic_t to refcount_t on hfi1_devdata->user_refcount

with the above kernel and rdma-core, the following messages will appear. "

[   54.214608] rdma_rxe: loaded
[   54.217089] infiniband rxe0: set active
[   54.217101] infiniband rxe0: added enp0s8
[  167.623200] rdma_rxe: cqe(32768) > max_cqe(32767)
[  167.645590] rdma_rxe: cqe(1) < current # elements in queue (6)
[  167.733297] rdma_rxe: cqe(32768) > max_cqe(32767)
[  169.074755] rdma_rxe: check_rkey: no MW matches rkey 0x1000247
[  169.074796] rdma_rxe: qp#27 moved to error state
[  169.138851] rdma_rxe: check_rkey: no MW matches rkey 0x10005de
[  169.138889] rdma_rxe: qp#30 moved to error state
[  169.160565] rdma_rxe: check_rkey: no MW matches rkey 0x10006f7
[  169.160601] rdma_rxe: qp#31 moved to error state
[  169.182132] rdma_rxe: check_rkey: no MW matches rkey 0x1000782
[  169.182170] rdma_rxe: qp#32 moved to error state
[  169.667803] rdma_rxe: check_rkey: no MR matches rkey 0x18d8
[  169.667850] rdma_rxe: qp#39 moved to error state
[  198.872649] rdma_rxe: cqe(32768) > max_cqe(32767)
[  198.894829] rdma_rxe: cqe(1) < current # elements in queue (6)
[  198.981839] rdma_rxe: cqe(32768) > max_cqe(32767)
[  200.332031] rdma_rxe: check_rkey: no MW matches rkey 0x1000887
[  200.332086] rdma_rxe: qp#58 moved to error state
[  200.396476] rdma_rxe: check_rkey: no MW matches rkey 0x1000b0d
[  200.396514] rdma_rxe: qp#61 moved to error state
[  200.417919] rdma_rxe: check_rkey: no MW matches rkey 0x1000c40
[  200.417956] rdma_rxe: qp#62 moved to error state
[  200.439616] rdma_rxe: check_rkey: no MW matches rkey 0x1000d24
[  200.439654] rdma_rxe: qp#63 moved to error state
[  200.933104] rdma_rxe: check_rkey: no MR matches rkey 0x37d8
[  200.933153] rdma_rxe: qp#70 moved to error state
[  206.880305] rdma_rxe: cqe(32768) > max_cqe(32767)
[  206.904030] rdma_rxe: cqe(1) < current # elements in queue (6)
[  206.991494] rdma_rxe: cqe(32768) > max_cqe(32767)
[  208.359987] rdma_rxe: check_rkey: no MW matches rkey 0x1000e4d
[  208.360028] rdma_rxe: qp#89 moved to error state
[  208.425637] rdma_rxe: check_rkey: no MW matches rkey 0x1001136
[  208.425675] rdma_rxe: qp#92 moved to error state
[  208.447333] rdma_rxe: check_rkey: no MW matches rkey 0x10012d8
[  208.447370] rdma_rxe: qp#93 moved to error state
[  208.469511] rdma_rxe: check_rkey: no MW matches rkey 0x100137a
[  208.469550] rdma_rxe: qp#94 moved to error state
[  208.956691] rdma_rxe: check_rkey: no MR matches rkey 0x5670
[  208.956731] rdma_rxe: qp#100 moved to error state
[  216.879703] rdma_rxe: cqe(32768) > max_cqe(32767)
[  216.902199] rdma_rxe: cqe(1) < current # elements in queue (6)
[  216.989264] rdma_rxe: cqe(32768) > max_cqe(32767)
[  218.363765] rdma_rxe: check_rkey: no MW matches rkey 0x10014d6
[  218.363808] rdma_rxe: qp#119 moved to error state
[  218.429474] rdma_rxe: check_rkey: no MW matches rkey 0x10017e4
[  218.429513] rdma_rxe: qp#122 moved to error state
[  218.451443] rdma_rxe: check_rkey: no MW matches rkey 0x1001895
[  218.451481] rdma_rxe: qp#123 moved to error state
[  218.473869] rdma_rxe: check_rkey: no MW matches rkey 0x1001910
[  218.473908] rdma_rxe: qp#124 moved to error state
[  218.963602] rdma_rxe: check_rkey: no MR matches rkey 0x757b
[  218.963641] rdma_rxe: qp#130 moved to error state
[  233.855140] rdma_rxe: cqe(32768) > max_cqe(32767)
[  233.877202] rdma_rxe: cqe(1) < current # elements in queue (6)
[  233.963952] rdma_rxe: cqe(32768) > max_cqe(32767)
[  235.305274] rdma_rxe: check_rkey: no MW matches rkey 0x1001ac2
[  235.305319] rdma_rxe: qp#149 moved to error state
[  235.368800] rdma_rxe: check_rkey: no MW matches rkey 0x1001db8
[  235.368838] rdma_rxe: qp#152 moved to error state
[  235.390155] rdma_rxe: check_rkey: no MW matches rkey 0x1001e4d
[  235.390192] rdma_rxe: qp#153 moved to error state
[  235.411336] rdma_rxe: check_rkey: no MW matches rkey 0x1001f4c
[  235.411374] rdma_rxe: qp#154 moved to error state
[  235.895784] rdma_rxe: check_rkey: no MR matches rkey 0x9482
[  235.895828] rdma_rxe: qp#161 moved to error state

"
Not sure if they are problems.
IMO, we should make further investigations.

Thanks
Zhu Yanjun

Quoted 2 lines

Thanks

#7 lore raw
Quoted 37 lines · Zhu Yanjun
Quoted 17 lines · Olga Kornievskaia
Quoted 16 lines · Zhu Yanjun
Quoted 15 lines · Leon Romanovsky
Quoted 14 lines · Zhu Yanjun
Quoted 11 lines · Zhu Yanjun
Quoted 5 lines · Leon Romanovsky

Hi,

Can you please help me to understand the RXE status in the upstream?

Does we still have crashes/interop issues/e.t.c?

I made some developments with the RXE in the upstream, from my usage with latest RXE,
I found the following:

1. rdma-core can not work well with latest RDMA git;

The latest RDMA git is
https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

"Latest" is a relative term, what SHA did you test?
Let's focus on fixing RXE before we will continue with new features.

Thanks a lot. I agree with you.

I believe simple rping still doesn't work linux-to-linux. The last working version (of rping in rxe) was 5.13 I think. I have posted a number of crashes rping encounters (gotta get that working before I can even try NFSoRDMA).

The following are my tests.

1. Modprobe rdma_rxe
2. Modprobe -v -r rdma_rxe
3. Rdma link add rxe
4. Rdma link del rxe
5. Latest rdma-core&& latest kernel upstream;

6. Latest kernel<  ------rping---->  5.10.y stable
7. Latest kernel<  ------rping---->  5.11.y stable
8. Latest kernel<  ------rping---->  5.12.y stable
9. Latest kernel<  ------rping---->  5.13.y stable
Quoted 5 lines

It seems that the latest kernel upstream (5.14-rc6) can rping other stable kernels.
Can you make tests again?

Zhu Yanjun

Hi,

I still get two simliar panic by rping or rdma_client/server on latest kernel vs 5.13:
Panic1:
--------------------------------------------------------

[  268.248642] BUG: unable to handle page fault for address: ffff9ae2c07a1414
[  268.251049] #PF: supervisor read access in kernel mode
[  268.252491] #PF: error_code(0x0000) - not-present page
[  268.253919] PGD 1000067 P4D 1000067 PUD 0
[  268.255052] Oops: 0000 [#1] SMP PTI
[  268.256055] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.14.0-rc6+ #1
[  268.257893] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014
[  268.259995] RIP: 0010:copy_data+0x2d/0x2a0 [rdma_rxe]
[  268.261114] Code: 00 00 41 57 41 56 41 55 41 54 55 53 48 83 ec 20 48 89 7c 24 08 89 74 24 10 44 89 4c 24 14 45 85 c0 0f 84 e8 00 00 00 45 89 c7<44>  8b 42 04 49 89 d6 44 89 44 24 18 45 39 f8 0f 8c 20 02 00 00 8b
[  268.265005] RSP: 0018:ffff9ae2404108b8 EFLAGS: 00010202
[  268.266145] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff8f7a8bf9da76
[  268.267703] RDX: ffff9ae2c07a1410 RSI: 0000000000000001 RDI: ffff8f7a71874400
[  268.269291] RBP: ffff8f7a482f87cc R08: 0000000000000010 R09: 0000000000000000
[  268.270871] R10: 00000000000000cb R11: 0000000000000001 R12: ffff8f7a482f8000
[  268.272468] R13: ffff8f7a8c038928 R14: ffff8f7a482f8008 R15: 0000000000000010
[  268.274080] FS:  0000000000000000(0000) GS:ffff8f7abec00000(0000) knlGS:0000000000000000
[  268.275899] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  268.277205] CR2: ffff9ae2c07a1414 CR3: 000000000263c002 CR4: 0000000000060ee0
[  268.278825] Call Trace:
[  268.279358]<IRQ>
[  268.279747]  rxe_responder+0x11b1/0x2490 [rdma_rxe]
[  268.280798]  rxe_do_task+0x9c/0xe0 [rdma_rxe]
[  268.281895]  rxe_rcv+0x286/0x8e0 [rdma_rxe]

...
------------------------------------------------------

Panic2:
--------------------------------------------------------

[  212.526854] BUG: unable to handle page fault for address: ffffbb97142acc14
[  212.530688] #PF: supervisor read access in kernel mode
[  212.533030] #PF: error_code(0x0000) - not-present page
[  212.535428] PGD 1000067 P4D 1000067 PUD 0
[  212.536970] Oops: 0000 [#1] SMP PTI
[  212.537748] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.14.0-rc6+ #1
[  212.538984] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014
[  212.540853] RIP: 0010:copy_data+0x2d/0x2a0 [rdma_rxe]
[  212.541957] Code: 00 00 41 57 41 56 41 55 41 54 55 53 48 83 ec 20 48 89 7c 24 08 89 74 24 10 44 89 4c 24 14 45 85 c0 0f 84 e8 00 00 00 45 89 c7<44>  8b 42 04 49 89 d6 44 89 44 24 18 45 39 f8 0f 8c 20 02 00 00 8b
[  212.546041] RSP: 0018:ffffbb9640410898 EFLAGS: 00010202
[  212.547200] RAX: ffffbb97142acc00 RBX: ffff95510ee6d000 RCX: ffff95510a802076
[  212.548782] RDX: ffffbb97142acc10 RSI: 0000000000000001 RDI: ffff95510ca00700
[  212.550369] RBP: 0000000000000010 R08: 0000000000000010 R09: 0000000000000000
[  212.551992] R10: 0000000000000001 R11: 0000000000000001 R12: ffff95510a802076
[  212.553613] R13: ffff9550f29acd28 R14: ffff95510ee6d008 R15: 0000000000000010
[  212.555225] FS:  0000000000000000(0000) GS:ffff95513ec00000(0000) knlGS:0000000000000000
[  212.556749] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  212.557846] CR2: ffffbb97142acc14 CR3: 0000000003b66005 CR4: 0000000000060ee0
[  212.559177] Call Trace:
[  212.559655]<IRQ>
[  212.560055]  send_data_in+0x55/0x73 [rdma_rxe]
[  212.560903]  rxe_responder.cold+0xea/0x1f8 [rdma_rxe]
[  212.561865]  rxe_do_task+0x9c/0xe0 [rdma_rxe]
[  212.562699]  rxe_rcv+0x286/0x8e0 [rdma_rxe]

...
------------------------------------------------------

Note: it is easy to reproduce the panic on the lastest kernel.

Best Regards,
Xiao Yang

Quoted 105 lines
Quoted 104 lines

Thank you for working on the code.

We (NFS community) do test NFSoRDMA every git pull using rxe and siw but lately have been encountering problems.

Quoted 99 lines

rdma-core:
313509f8 (HEAD -> master, origin/master, origin/HEAD) Merge pull request #1038 from selvintxavier/master
2d3dc48b Merge pull request #1039 from amzn/pyverbs-mac-fix-pr 327d45e0 tests: Add missing MAC element to args list 66aba73d bnxt_re/lib: Move hardware queue to 16B aligned indices 8754fb51 bnxt_re/lib: Use separate indices for shadow queue be4d8abf bnxt_re/lib: add a function to initialize software queue

kernel rdma:
0050a57638ca (HEAD -> for-next, origin/for-next, origin/HEAD) RDMA/qedr: Improve error logs for rdma_alloc_tid error return 090473004b02 RDMA/qed: Use accurate error num in qed_cxt_dynamic_ilt_alloc 991c4274dc17 RDMA/hfi1: Fix typo in comments
8d7e415d5561 docs: Fix infiniband uverbs minor number bbafcbc2b1c9 RDMA/iwpm: Rely on the rdma_nl_[un]register() to ensure that requests are valid
bdb0e4e3ff19 RDMA/iwpm: Remove not-needed reference counting e677b72a0647 RDMA/iwcm: Release resources if iw_cm module initialization fails a0293eb24936 RDMA/hfi1: Convert from atomic_t to refcount_t on hfi1_devdata->user_refcount

with the above kernel and rdma-core, the following messages will appear. "

[   54.214608] rdma_rxe: loaded
[   54.217089] infiniband rxe0: set active
[   54.217101] infiniband rxe0: added enp0s8
[  167.623200] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  167.645590] rdma_rxe: cqe(1)<  current # elements in queue (6)
[  167.733297] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  169.074755] rdma_rxe: check_rkey: no MW matches rkey 0x1000247
[  169.074796] rdma_rxe: qp#27 moved to error state
[  169.138851] rdma_rxe: check_rkey: no MW matches rkey 0x10005de
[  169.138889] rdma_rxe: qp#30 moved to error state
[  169.160565] rdma_rxe: check_rkey: no MW matches rkey 0x10006f7
[  169.160601] rdma_rxe: qp#31 moved to error state
[  169.182132] rdma_rxe: check_rkey: no MW matches rkey 0x1000782
[  169.182170] rdma_rxe: qp#32 moved to error state
[  169.667803] rdma_rxe: check_rkey: no MR matches rkey 0x18d8
[  169.667850] rdma_rxe: qp#39 moved to error state
[  198.872649] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  198.894829] rdma_rxe: cqe(1)<  current # elements in queue (6)
[  198.981839] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  200.332031] rdma_rxe: check_rkey: no MW matches rkey 0x1000887
[  200.332086] rdma_rxe: qp#58 moved to error state
[  200.396476] rdma_rxe: check_rkey: no MW matches rkey 0x1000b0d
[  200.396514] rdma_rxe: qp#61 moved to error state
[  200.417919] rdma_rxe: check_rkey: no MW matches rkey 0x1000c40
[  200.417956] rdma_rxe: qp#62 moved to error state
[  200.439616] rdma_rxe: check_rkey: no MW matches rkey 0x1000d24
[  200.439654] rdma_rxe: qp#63 moved to error state
[  200.933104] rdma_rxe: check_rkey: no MR matches rkey 0x37d8
[  200.933153] rdma_rxe: qp#70 moved to error state
[  206.880305] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  206.904030] rdma_rxe: cqe(1)<  current # elements in queue (6)
[  206.991494] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  208.359987] rdma_rxe: check_rkey: no MW matches rkey 0x1000e4d
[  208.360028] rdma_rxe: qp#89 moved to error state
[  208.425637] rdma_rxe: check_rkey: no MW matches rkey 0x1001136
[  208.425675] rdma_rxe: qp#92 moved to error state
[  208.447333] rdma_rxe: check_rkey: no MW matches rkey 0x10012d8
[  208.447370] rdma_rxe: qp#93 moved to error state
[  208.469511] rdma_rxe: check_rkey: no MW matches rkey 0x100137a
[  208.469550] rdma_rxe: qp#94 moved to error state
[  208.956691] rdma_rxe: check_rkey: no MR matches rkey 0x5670
[  208.956731] rdma_rxe: qp#100 moved to error state
[  216.879703] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  216.902199] rdma_rxe: cqe(1)<  current # elements in queue (6)
[  216.989264] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  218.363765] rdma_rxe: check_rkey: no MW matches rkey 0x10014d6
[  218.363808] rdma_rxe: qp#119 moved to error state
[  218.429474] rdma_rxe: check_rkey: no MW matches rkey 0x10017e4
[  218.429513] rdma_rxe: qp#122 moved to error state
[  218.451443] rdma_rxe: check_rkey: no MW matches rkey 0x1001895
[  218.451481] rdma_rxe: qp#123 moved to error state
[  218.473869] rdma_rxe: check_rkey: no MW matches rkey 0x1001910
[  218.473908] rdma_rxe: qp#124 moved to error state
[  218.963602] rdma_rxe: check_rkey: no MR matches rkey 0x757b
[  218.963641] rdma_rxe: qp#130 moved to error state
[  233.855140] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  233.877202] rdma_rxe: cqe(1)<  current # elements in queue (6)
[  233.963952] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  235.305274] rdma_rxe: check_rkey: no MW matches rkey 0x1001ac2
[  235.305319] rdma_rxe: qp#149 moved to error state
[  235.368800] rdma_rxe: check_rkey: no MW matches rkey 0x1001db8
[  235.368838] rdma_rxe: qp#152 moved to error state
[  235.390155] rdma_rxe: check_rkey: no MW matches rkey 0x1001e4d
[  235.390192] rdma_rxe: qp#153 moved to error state
[  235.411336] rdma_rxe: check_rkey: no MW matches rkey 0x1001f4c
[  235.411374] rdma_rxe: qp#154 moved to error state
[  235.895784] rdma_rxe: check_rkey: no MR matches rkey 0x9482
[  235.895828] rdma_rxe: qp#161 moved to error state

"
Not sure if they are problems.
IMO, we should make further investigations.

Thanks
Zhu Yanjun

Quoted 1 line

Thanks

#8 lore raw
Quoted 101 lines · On Wed, Aug 18, 2021 at 2:43 PM [email protected] <[email protected]>…
Quoted 37 lines · Zhu Yanjun
Quoted 17 lines · Olga Kornievskaia
Quoted 16 lines · Zhu Yanjun
Quoted 15 lines · Leon Romanovsky
Quoted 14 lines · Zhu Yanjun
Quoted 11 lines · Zhu Yanjun
Quoted 5 lines · Leon Romanovsky

Hi,

Can you please help me to understand the RXE status in the upstream?

Does we still have crashes/interop issues/e.t.c?

I made some developments with the RXE in the upstream, from my usage with latest RXE,
I found the following:

1. rdma-core can not work well with latest RDMA git;

The latest RDMA git is
https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

"Latest" is a relative term, what SHA did you test?
Let's focus on fixing RXE before we will continue with new features.

Thanks a lot. I agree with you.

I believe simple rping still doesn't work linux-to-linux. The last working version (of rping in rxe) was 5.13 I think. I have posted a number of crashes rping encounters (gotta get that working before I can even try NFSoRDMA).

The following are my tests.

1. Modprobe rdma_rxe
2. Modprobe -v -r rdma_rxe
3. Rdma link add rxe
4. Rdma link del rxe
5. Latest rdma-core&& latest kernel upstream;

6. Latest kernel<  ------rping---->  5.10.y stable
7. Latest kernel<  ------rping---->  5.11.y stable
8. Latest kernel<  ------rping---->  5.12.y stable
9. Latest kernel<  ------rping---->  5.13.y stable
Quoted 5 lines

It seems that the latest kernel upstream (5.14-rc6) can rping other stable kernels.
Can you make tests again?

Zhu Yanjun

Hi,

I still get two simliar panic by rping or rdma_client/server on latest kernel vs 5.13:
Panic1:
--------------------------------------------------------

[  268.248642] BUG: unable to handle page fault for address: ffff9ae2c07a1414
[  268.251049] #PF: supervisor read access in kernel mode
[  268.252491] #PF: error_code(0x0000) - not-present page
[  268.253919] PGD 1000067 P4D 1000067 PUD 0
[  268.255052] Oops: 0000 [#1] SMP PTI
[  268.256055] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.14.0-rc6+ #1
[  268.257893] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014
[  268.259995] RIP: 0010:copy_data+0x2d/0x2a0 [rdma_rxe]
[  268.261114] Code: 00 00 41 57 41 56 41 55 41 54 55 53 48 83 ec 20 48 89 7c 24 08 89 74 24 10 44 89 4c 24 14 45 85 c0 0f 84 e8 00 00 00 45 89 c7<44>  8b 42 04 49 89 d6 44 89 44 24 18 45 39 f8 0f 8c 20 02 00 00 8b
[  268.265005] RSP: 0018:ffff9ae2404108b8 EFLAGS: 00010202
[  268.266145] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff8f7a8bf9da76
[  268.267703] RDX: ffff9ae2c07a1410 RSI: 0000000000000001 RDI: ffff8f7a71874400
[  268.269291] RBP: ffff8f7a482f87cc R08: 0000000000000010 R09: 0000000000000000
[  268.270871] R10: 00000000000000cb R11: 0000000000000001 R12: ffff8f7a482f8000
[  268.272468] R13: ffff8f7a8c038928 R14: ffff8f7a482f8008 R15: 0000000000000010
[  268.274080] FS:  0000000000000000(0000) GS:ffff8f7abec00000(0000) knlGS:0000000000000000
[  268.275899] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  268.277205] CR2: ffff9ae2c07a1414 CR3: 000000000263c002 CR4: 0000000000060ee0
[  268.278825] Call Trace:
[  268.279358]<IRQ>
[  268.279747]  rxe_responder+0x11b1/0x2490 [rdma_rxe]
[  268.280798]  rxe_do_task+0x9c/0xe0 [rdma_rxe]
[  268.281895]  rxe_rcv+0x286/0x8e0 [rdma_rxe]

...
------------------------------------------------------

Panic2:
--------------------------------------------------------

[  212.526854] BUG: unable to handle page fault for address: ffffbb97142acc14
[  212.530688] #PF: supervisor read access in kernel mode
[  212.533030] #PF: error_code(0x0000) - not-present page
[  212.535428] PGD 1000067 P4D 1000067 PUD 0
[  212.536970] Oops: 0000 [#1] SMP PTI
[  212.537748] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.14.0-rc6+ #1
[  212.538984] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.12.0-2.fc30 04/01/2014
[  212.540853] RIP: 0010:copy_data+0x2d/0x2a0 [rdma_rxe]
[  212.541957] Code: 00 00 41 57 41 56 41 55 41 54 55 53 48 83 ec 20 48 89 7c 24 08 89 74 24 10 44 89 4c 24 14 45 85 c0 0f 84 e8 00 00 00 45 89 c7<44>  8b 42 04 49 89 d6 44 89 44 24 18 45 39 f8 0f 8c 20 02 00 00 8b
[  212.546041] RSP: 0018:ffffbb9640410898 EFLAGS: 00010202
[  212.547200] RAX: ffffbb97142acc00 RBX: ffff95510ee6d000 RCX: ffff95510a802076
[  212.548782] RDX: ffffbb97142acc10 RSI: 0000000000000001 RDI: ffff95510ca00700
[  212.550369] RBP: 0000000000000010 R08: 0000000000000010 R09: 0000000000000000
[  212.551992] R10: 0000000000000001 R11: 0000000000000001 R12: ffff95510a802076
[  212.553613] R13: ffff9550f29acd28 R14: ffff95510ee6d008 R15: 0000000000000010
[  212.555225] FS:  0000000000000000(0000) GS:ffff95513ec00000(0000) knlGS:0000000000000000
[  212.556749] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  212.557846] CR2: ffffbb97142acc14 CR3: 0000000003b66005 CR4: 0000000000060ee0
[  212.559177] Call Trace:
[  212.559655]<IRQ>
[  212.560055]  send_data_in+0x55/0x73 [rdma_rxe]
[  212.560903]  rxe_responder.cold+0xea/0x1f8 [rdma_rxe]
[  212.561865]  rxe_do_task+0x9c/0xe0 [rdma_rxe]
[  212.562699]  rxe_rcv+0x286/0x8e0 [rdma_rxe]
Quoted 4 lines

...
------------------------------------------------------

Note: it is easy to reproduce the panic on the lastest kernel.

Can you let me know how to reproduce the panic?

1. linux upstream < ----rping---- > linux upstream?
2. just run rping?
3. how do you create rxe? with rdma link or rxe_cfg?
4. do you make other operations?
5. other operations?

Thanks.
Zhu Yanjun

Quoted 111 lines

Best Regards,
Xiao Yang

Quoted 105 lines
Quoted 104 lines

Thank you for working on the code.

We (NFS community) do test NFSoRDMA every git pull using rxe and siw but lately have been encountering problems.

Quoted 99 lines

rdma-core:
313509f8 (HEAD -> master, origin/master, origin/HEAD) Merge pull request #1038 from selvintxavier/master
2d3dc48b Merge pull request #1039 from amzn/pyverbs-mac-fix-pr 327d45e0 tests: Add missing MAC element to args list 66aba73d bnxt_re/lib: Move hardware queue to 16B aligned indices 8754fb51 bnxt_re/lib: Use separate indices for shadow queue be4d8abf bnxt_re/lib: add a function to initialize software queue

kernel rdma:
0050a57638ca (HEAD -> for-next, origin/for-next, origin/HEAD) RDMA/qedr: Improve error logs for rdma_alloc_tid error return 090473004b02 RDMA/qed: Use accurate error num in qed_cxt_dynamic_ilt_alloc 991c4274dc17 RDMA/hfi1: Fix typo in comments
8d7e415d5561 docs: Fix infiniband uverbs minor number bbafcbc2b1c9 RDMA/iwpm: Rely on the rdma_nl_[un]register() to ensure that requests are valid
bdb0e4e3ff19 RDMA/iwpm: Remove not-needed reference counting e677b72a0647 RDMA/iwcm: Release resources if iw_cm module initialization fails a0293eb24936 RDMA/hfi1: Convert from atomic_t to refcount_t on hfi1_devdata->user_refcount

with the above kernel and rdma-core, the following messages will appear. "

[   54.214608] rdma_rxe: loaded
[   54.217089] infiniband rxe0: set active
[   54.217101] infiniband rxe0: added enp0s8
[  167.623200] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  167.645590] rdma_rxe: cqe(1)<  current # elements in queue (6)
[  167.733297] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  169.074755] rdma_rxe: check_rkey: no MW matches rkey 0x1000247
[  169.074796] rdma_rxe: qp#27 moved to error state
[  169.138851] rdma_rxe: check_rkey: no MW matches rkey 0x10005de
[  169.138889] rdma_rxe: qp#30 moved to error state
[  169.160565] rdma_rxe: check_rkey: no MW matches rkey 0x10006f7
[  169.160601] rdma_rxe: qp#31 moved to error state
[  169.182132] rdma_rxe: check_rkey: no MW matches rkey 0x1000782
[  169.182170] rdma_rxe: qp#32 moved to error state
[  169.667803] rdma_rxe: check_rkey: no MR matches rkey 0x18d8
[  169.667850] rdma_rxe: qp#39 moved to error state
[  198.872649] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  198.894829] rdma_rxe: cqe(1)<  current # elements in queue (6)
[  198.981839] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  200.332031] rdma_rxe: check_rkey: no MW matches rkey 0x1000887
[  200.332086] rdma_rxe: qp#58 moved to error state
[  200.396476] rdma_rxe: check_rkey: no MW matches rkey 0x1000b0d
[  200.396514] rdma_rxe: qp#61 moved to error state
[  200.417919] rdma_rxe: check_rkey: no MW matches rkey 0x1000c40
[  200.417956] rdma_rxe: qp#62 moved to error state
[  200.439616] rdma_rxe: check_rkey: no MW matches rkey 0x1000d24
[  200.439654] rdma_rxe: qp#63 moved to error state
[  200.933104] rdma_rxe: check_rkey: no MR matches rkey 0x37d8
[  200.933153] rdma_rxe: qp#70 moved to error state
[  206.880305] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  206.904030] rdma_rxe: cqe(1)<  current # elements in queue (6)
[  206.991494] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  208.359987] rdma_rxe: check_rkey: no MW matches rkey 0x1000e4d
[  208.360028] rdma_rxe: qp#89 moved to error state
[  208.425637] rdma_rxe: check_rkey: no MW matches rkey 0x1001136
[  208.425675] rdma_rxe: qp#92 moved to error state
[  208.447333] rdma_rxe: check_rkey: no MW matches rkey 0x10012d8
[  208.447370] rdma_rxe: qp#93 moved to error state
[  208.469511] rdma_rxe: check_rkey: no MW matches rkey 0x100137a
[  208.469550] rdma_rxe: qp#94 moved to error state
[  208.956691] rdma_rxe: check_rkey: no MR matches rkey 0x5670
[  208.956731] rdma_rxe: qp#100 moved to error state
[  216.879703] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  216.902199] rdma_rxe: cqe(1)<  current # elements in queue (6)
[  216.989264] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  218.363765] rdma_rxe: check_rkey: no MW matches rkey 0x10014d6
[  218.363808] rdma_rxe: qp#119 moved to error state
[  218.429474] rdma_rxe: check_rkey: no MW matches rkey 0x10017e4
[  218.429513] rdma_rxe: qp#122 moved to error state
[  218.451443] rdma_rxe: check_rkey: no MW matches rkey 0x1001895
[  218.451481] rdma_rxe: qp#123 moved to error state
[  218.473869] rdma_rxe: check_rkey: no MW matches rkey 0x1001910
[  218.473908] rdma_rxe: qp#124 moved to error state
[  218.963602] rdma_rxe: check_rkey: no MR matches rkey 0x757b
[  218.963641] rdma_rxe: qp#130 moved to error state
[  233.855140] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  233.877202] rdma_rxe: cqe(1)<  current # elements in queue (6)
[  233.963952] rdma_rxe: cqe(32768)>  max_cqe(32767)
[  235.305274] rdma_rxe: check_rkey: no MW matches rkey 0x1001ac2
[  235.305319] rdma_rxe: qp#149 moved to error state
[  235.368800] rdma_rxe: check_rkey: no MW matches rkey 0x1001db8
[  235.368838] rdma_rxe: qp#152 moved to error state
[  235.390155] rdma_rxe: check_rkey: no MW matches rkey 0x1001e4d
[  235.390192] rdma_rxe: qp#153 moved to error state
[  235.411336] rdma_rxe: check_rkey: no MW matches rkey 0x1001f4c
[  235.411374] rdma_rxe: qp#154 moved to error state
[  235.895784] rdma_rxe: check_rkey: no MR matches rkey 0x9482
[  235.895828] rdma_rxe: qp#161 moved to error state

"
Not sure if they are problems.
IMO, we should make further investigations.

Thanks
Zhu Yanjun

Quoted 1 line

Thanks

#9 lore raw
Quoted 3 lines · Zhu Yanjun

Can you let me know how to reproduce the panic?

1. linux upstream< ----rping----> linux upstream?

rdma_client on v5.13< ---> rdma_server on upstream kernel.

Quoted 1 line

2. just run rping?

Running rdma_client on v5.13 and rdma_server on upstream can reproduce the issue.

Note: running rping can reproduce the issue as well.

Quoted 1 line

3. how do you create rxe? with rdma link or rxe_cfg?

rdma link add

Quoted 1 line

4. do you make other operations?

No

Quoted 1 line

5. other operations?

No

Quoted 3 lines

Thanks.
Zhu Yanjun

#10 lore raw
Quoted 4 lines · On Wed, Aug 18, 2021 at 3:44 PM [email protected] <[email protected]>…
Quoted 2 lines · Zhu Yanjun

Can you let me know how to reproduce the panic?

Quoted 8 lines
Quoted 1 line

1. linux upstream< ----rping----> linux upstream?

rdma_client on v5.13< ---> rdma_server on upstream kernel.

Quoted 1 line

2. just run rping?

Running rdma_client on v5.13 and rdma_server on upstream can reproduce the issue.

Note: running rping can reproduce the issue as well.

rping and rdma_server/rdma_client are from the latest rdma-core?

Thanks
Zhu Yanjun

Quoted 9 lines
Quoted 1 line

3. how do you create rxe? with rdma link or rxe_cfg?

rdma link add

Quoted 1 line

4. do you make other operations?

No

Quoted 1 line

5. other operations?

No

Quoted 3 lines

Thanks.
Zhu Yanjun

#11 lore raw
Quoted 6 lines · Zhu Yanjun
Quoted 4 lines · On Wed, Aug 18, 2021 at 3:44 PM [email protected] <[email protected]>…
Quoted 3 lines · Zhu Yanjun

Can you let me know how to reproduce the panic?

1. linux upstream< ----rping----> linux upstream?

Quoted 8 lines
Quoted 7 lines

rdma_client on v5.13< ---> rdma_server on upstream kernel.

Quoted 1 line

2. just run rping?

Running rdma_client on v5.13 and rdma_server on upstream can reproduce the issue.

Note: running rping can reproduce the issue as well.

rping and rdma_server/rdma_client are from the latest rdma-core?

Yes, use the latest rdma-core from
https://github.com/linux-rdma/rdma-core (master branch).

Quoted 12 lines

Thanks
Zhu Yanjun

Quoted 9 lines
Quoted 1 line

3. how do you create rxe? with rdma link or rxe_cfg?

rdma link add

Quoted 1 line

4. do you make other operations?

No

Quoted 1 line

5. other operations?

No

Quoted 3 lines

Thanks.
Zhu Yanjun

#12 lore raw
Quoted 10 lines · On Wed, Aug 18, 2021 at 10:33 PM [email protected] <[email protected]>…
Quoted 8 lines · Zhu Yanjun
Quoted 6 lines · On Wed, Aug 18, 2021 at 3:44 PM [email protected] <[email protected]>…
Quoted 3 lines · Zhu Yanjun

Can you let me know how to reproduce the panic?

1. linux upstream< ----rping----> linux upstream?

rdma_client on v5.13< ---> rdma_server on upstream kernel.

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 1 line

2. just run rping?

Running rdma_client on v5.13 and rdma_server on upstream can reproduce the issue.

Note: running rping can reproduce the issue as well.

rping and rdma_server/rdma_client are from the latest rdma-core?

Yes, use the latest rdma-core from
https://github.com/linux-rdma/rdma-core (master branch).

Latest kernel + latest rdma-core < ------rping---- > 5.10.y stable + latest rdma-core
Latest kernel + latest rdma-core < ------rping---- > 5.11.y stable + latest rdma-core
Latest kernel + latest rdma-core < ------rping---- > 5.12.y stable + latest rdma-core
Latest kernel + latest rdma-core < ------rping---- > 5.13.y stable + latest rdma-core

The above works well.

Zhu Yanjun

Quoted 12 lines
Quoted 12 lines

Thanks
Zhu Yanjun

Quoted 9 lines
Quoted 1 line

3. how do you create rxe? with rdma link or rxe_cfg?

rdma link add

Quoted 1 line

4. do you make other operations?

No

Quoted 1 line

5. other operations?

No

Quoted 3 lines

Thanks.
Zhu Yanjun

#13 lore raw
Quoted 10 lines · Zhu Yanjun

Latest kernel + latest rdma-coOnre< ------rping----> 5.10.y stable + latest rdma-core
Latest kernel + latest rdma-core< ------rping----> 5.11.y stable + latest rdma-core
Latest kernel + latest rdma-core< ------rping----> 5.12.y stable + latest rdma-core
Latest kernel + latest rdma-core< ------rping----> 5.13.y stable + latest rdma-core

The above works well.

Hi Yanjun,

Sorry, I don't know why you cannot reproduce the bug.

Did you see the similar bug reported by Olga Kornievskaia?
https://www.spinics.net/lists/linux-rdma/msg104358.html
https://www.spinics.net/lists/linux-rdma/msg104359.html
https://www.spinics.net/lists/linux-rdma/msg104360.html

Best Regards,
Xiao Yang

Quoted 2 lines

Zhu Yanjun

#14 lore raw
Quoted 15 lines · On 8/20/21 2:42 AM, [email protected] wrote:
Quoted 10 lines · Zhu Yanjun

Latest kernel + latest rdma-coOnre< ------rping----> 5.10.y stable + latest rdma-core
Latest kernel + latest rdma-core< ------rping----> 5.11.y stable + latest rdma-core
Latest kernel + latest rdma-core< ------rping----> 5.12.y stable + latest rdma-core
Latest kernel + latest rdma-core< ------rping----> 5.13.y stable + latest rdma-core

The above works well.

Hi Yanjun,

Sorry, I don't know why you cannot reproduce the bug.

Quoted 9 lines

Did you see the similar bug reported by Olga Kornievskaia?
https://www.spinics.net/lists/linux-rdma/msg104358.html
https://www.spinics.net/lists/linux-rdma/msg104359.html
https://www.spinics.net/lists/linux-rdma/msg104360.html

Best Regards,
Xiao Yang

Quoted 2 lines

Zhu Yanjun

There is some interest in the current status of rping on rxe.
I have looked at several configurations and tested the following test cases:

1. The python test suite in rdma-core
2. ib_xxx_bw and ib_xxx_bw -R for RC
3. rping

Between the following node configurations.

A. 5.11.0 (ubuntu 21.04 OOB) + rdma-core 33.1 (ubuntu 21.04 OOB)
B. 5.11.0 + current rdma-core
+ "Provider/rxe:Set the correct value of resid for inline data" (a.k.a rdma-core+)
C. 5.14.0-rc1+ (for-next current)
+ 5 recent bug fixes (a.k.a. for-next+)
RDMA/rxe:Fix bug in get srq wqe in rxe_resp.c.patch

RDMA/rxe:Fix bug in rxe_net.c.patch

RDMA/rxe:Add memory barriers to kernel queues.patch

RDMA/rxe:Fix memory allocation while locked.patch

RDMA/rxe:Zero out index member of struct rxe_queue.patch
+ rdma-core+
D. for-next+ + rdma-core (33.1)

Results:

	1.  A N/A
	1.  B no errors, some skips
	1.  C no errors, some skips
	1.  D N/A

(n.b. requires adding IPV6 address == gid[0] by hand)

2. [A-D] -> [A-D] all pass

3. A -> A, C -> C, D -> D all pass, all other combinations fail

(RDMA_resolve_route: No such device. Not yet sure cause of failures but looking into it.) In theory these should all work but rdmacm is more sensitive to configuration than verbs.

Bob

#15 lore raw
Quoted 59 lines · Bob Pearson
Quoted 15 lines · On 8/20/21 2:42 AM, [email protected] wrote:
Quoted 10 lines · Zhu Yanjun

Latest kernel + latest rdma-coOnre< ------rping----> 5.10.y stable + latest rdma-core
Latest kernel + latest rdma-core< ------rping----> 5.11.y stable + latest rdma-core
Latest kernel + latest rdma-core< ------rping----> 5.12.y stable + latest rdma-core
Latest kernel + latest rdma-core< ------rping----> 5.13.y stable + latest rdma-core

The above works well.

Hi Yanjun,

Sorry, I don't know why you cannot reproduce the bug.

Quoted 9 lines

Did you see the similar bug reported by Olga Kornievskaia?
https://www.spinics.net/lists/linux-rdma/msg104358.html
https://www.spinics.net/lists/linux-rdma/msg104359.html
https://www.spinics.net/lists/linux-rdma/msg104360.html

Best Regards,
Xiao Yang

Quoted 2 lines

Zhu Yanjun

There is some interest in the current status of rping on rxe.
I have looked at several configurations and tested the following test cases:

1. The python test suite in rdma-core
2. ib_xxx_bw and ib_xxx_bw -R for RC
3. rping

Between the following node configurations.

A. 5.11.0 (ubuntu 21.04 OOB) + rdma-core 33.1 (ubuntu 21.04 OOB)
B. 5.11.0 + current rdma-core
+ "Provider/rxe:Set the correct value of resid for inline data" (a.k.a rdma-core+)
C. 5.14.0-rc1+ (for-next current)
+ 5 recent bug fixes (a.k.a. for-next+)
RDMA/rxe:Fix bug in get srq wqe in rxe_resp.c.patch

RDMA/rxe:Fix bug in rxe_net.c.patch

RDMA/rxe:Add memory barriers to kernel queues.patch

RDMA/rxe:Fix memory allocation while locked.patch

RDMA/rxe:Zero out index member of struct rxe_queue.patch
+ rdma-core+
D. for-next+ + rdma-core (33.1)

Results:

	1.  A N/A
	1.  B no errors, some skips
	1.  C no errors, some skips
	1.  D N/A

(n.b. requires adding IPV6 address == gid[0] by hand)

Quoted 9 lines

2. [A-D] -> [A-D] all pass

3. A -> A, C -> C, D -> D all pass, all other combinations fail

(RDMA_resolve_route: No such device. Not yet sure cause of failures but looking into it.) In theory these should all work but rdmacm is more sensitive to configuration than verbs.

Bob

Found the problem (thank you google) If you run both

server$ rping -s -a nn.nn.nn.nn
client$ rping -c -a nn.nn.nn.nn

now all tests pass for rping as well.

Bob

#18 lore raw
In reply to Leon Romanovsky
Quoted 4 lines · Leon Romanovsky

Hi,

Can you please help me to understand the RXE status in the upstream?

Hi, all

On the Ubuntu 20.04, kernel: 5.4.0-80, with the latest rdma-core,

"

root@xxx:~/rdma-core# cat /etc/issue
Ubuntu 20.04.2 LTS \n \l
root@xxx:~/rdma-core# uname -a
Linux 5.4.0-80-generic #90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021
x86_64 x86_64 x86_64 GNU/Linux
root@xxx:~/rdma-core# git log -1 --oneline
206a0cfd (HEAD -> master, origin/master, origin/HEAD) Merge pull
request #1047 from yishaih/mlx5_misc
"

Run run_tests.py, I got the following errors.

Not sure if it is a problem. Please comment on it.

It is easy to reproduce on Ubuntu 20.04 + 5.4.0-80.

" .............sssssssss..FFF........sssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.ssssssssssssssssssssssssss....s...ss..........s....s..s.......ssReceived the following exceptions: {'active': BrokenBarrierError()}
EReceived the following exceptions: {'active': BrokenBarrierError()}
E........ss
======================================================================
ERROR: test_rdmacm_async_ex_multicast_traffic (tests.test_rdmacm.CMTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/root/rdma-core/tests/utils.py", line 976, in inner
return func(instance)
File "/root/rdma-core/tests/test_rdmacm.py", line 42, in
test_rdmacm_async_ex_multicast_traffic
self.two_nodes_rdmacm_traffic(CMAsyncConnection,
File "/root/rdma-core/tests/base.py", line 368, in two_nodes_rdmacm_traffic
raise(res)
threading.BrokenBarrierError

======================================================================
ERROR: test_rdmacm_async_multicast_traffic (tests.test_rdmacm.CMTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/root/rdma-core/tests/utils.py", line 976, in inner
return func(instance)
File "/root/rdma-core/tests/test_rdmacm.py", line 36, in
test_rdmacm_async_multicast_traffic
self.two_nodes_rdmacm_traffic(CMAsyncConnection,
File "/root/rdma-core/tests/base.py", line 368, in two_nodes_rdmacm_traffic
raise(res)
threading.BrokenBarrierError

======================================================================
FAIL: test_phys_port_cnt_ex (tests.test_device.DeviceTest) Test phys_port_cnt_ex
----------------------------------------------------------------------
Traceback (most recent call last):
File "/root/rdma-core/tests/test_device.py", line 222, in
test_phys_port_cnt_ex
self.assertEqual(phys_port_cnt, phys_port_cnt_ex,
AssertionError: 1 != 0 : phys_port_cnt_ex and phys_port_cnt should be equal if number of ports is less than 256

======================================================================
FAIL: test_query_device (tests.test_device.DeviceTest)
Test ibv_query_device()
----------------------------------------------------------------------
Traceback (most recent call last):
File "/root/rdma-core/tests/test_device.py", line 63, in test_query_device
self.verify_device_attr(attr, dev)
File "/root/rdma-core/tests/test_device.py", line 187, in verify_device_attr

    assert attr.vendor_id != 0

AssertionError

======================================================================
FAIL: test_query_device_ex (tests.test_device.DeviceTest)
Test ibv_query_device_ex()
----------------------------------------------------------------------
Traceback (most recent call last):
File "/root/rdma-core/tests/test_device.py", line 206, in test_query_device_ex
self.verify_device_attr(attr_ex.orig_attr, dev)
File "/root/rdma-core/tests/test_device.py", line 187, in verify_device_attr

    assert attr.vendor_id != 0

AssertionError

----------------------------------------------------------------------
Ran 205 tests in 40.112s

FAILED (failures=3, errors=2, skipped=137)
Traceback (most recent call last):
File "device.pyx", line 170, in pyverbs.device.Context.close
AttributeError: 'NoneType' object has no attribute 'debug' Exception ignored in: 'pyverbs.device.Context.__dealloc__' Traceback (most recent call last):
File "device.pyx", line 170, in pyverbs.device.Context.close
AttributeError: 'NoneType' object has no attribute 'debug' "

Quoted 7 lines

Does we still have crashes/interop issues/e.t.c?

Latest commit is:
20da44dfe8ef ("RDMA/mlx5: Drop in-driver verbs object creations")

Thanks