Skip to contentSkip to conversation

New: RDMA Mellanox RoCE drivers are unresponsive to ARP updates during a reconnect

Bug 214523 [email protected] 6 participants 10 messages Sep 24, 2021 – Oct 15, 2021 InfiniBand

lore
Share & tools
lore
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=214523

Bug ID: 214523
Summary: RDMA Mellanox RoCE drivers are unresponsive to ARP
updates during a reconnect
Product: Drivers
Version: 2.5
Kernel Version: 5.14
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Infiniband/RDMA
Assignee: [email protected]
Reporter: [email protected]
Regression: No

RoCE RDMA connection uses CMA protocol to establish an RDMA connection. During the setup the code uses hard coded timeout/retry values. These values are used for when Connect Request is not being answered to to re-try the request. During the re-try attempts the ARP updates of the destination server are ignored. Current timeout values lead to 4+minutes long attempt at connecting to a server that no longer owns the IP since the ARP update happens.

The ask is to make the timeout/retry values configurable via procfs or sysfs. This will allow for environments that use RoCE to reduce the timeouts to a more reasonable values and be able to react to the ARP updates faster. Other CMA users (eg IB or others) can continue to use existing values.

The problem exist in all kernel versions but bugzilla is filed for 5.14 kernel.

The use case is (RoCE-based) NFSoRDMA where a server went down and another server was brought up in its place. RDMA layer introduces 4+ minutes in being able to re-establish an RDMA connection and let IO resume, due to inability to react to the ARP update.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#1 lore raw
Quoted 31 lines · On Fri, Sep 24, 2021 at 03:34:32PM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=214523

Bug ID: 214523
Summary: RDMA Mellanox RoCE drivers are unresponsive to ARP
updates during a reconnect
Product: Drivers
Version: 2.5
Kernel Version: 5.14
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Infiniband/RDMA
Assignee: [email protected]
Reporter: [email protected]
Regression: No

RoCE RDMA connection uses CMA protocol to establish an RDMA connection. During the setup the code uses hard coded timeout/retry values. These values are used for when Connect Request is not being answered to to re-try the request. During the re-try attempts the ARP updates of the destination server are ignored. Current timeout values lead to 4+minutes long attempt at connecting to a server that no longer owns the IP since the ARP update happens.

The ask is to make the timeout/retry values configurable via procfs or sysfs. This will allow for environments that use RoCE to reduce the timeouts to a more reasonable values and be able to react to the ARP updates faster. Other CMA users (eg IB or others) can continue to use existing values.

Quoted 6 lines

The problem exist in all kernel versions but bugzilla is filed for 5.14 kernel.

The use case is (RoCE-based) NFSoRDMA where a server went down and another server was brought up in its place. RDMA layer introduces 4+ minutes in being able to re-establish an RDMA connection and let IO resume, due to inability to react to the ARP update.

RDMA-CM has many different timeouts, so I hope that my answer is for the right timeout.

We probably need to extend rdma_connect() to receive remote_cm_response_timeout value, so NFSoRDMA will set it to whatever value its appropriate.

The timewait will be calculated based it in ib_send_cm_req().

Thanks

Quoted 6 lines
Signature · 5 lines
-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.
#2 lore raw

Hi Leon-

Thanks for the suggestion! More below.

Quoted 29 lines · Leon Romanovsky
Quoted 26 lines · On Fri, Sep 24, 2021 at 03:34:32PM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=214523

Bug ID: 214523
Summary: RDMA Mellanox RoCE drivers are unresponsive to ARP
updates during a reconnect
Product: Drivers
Version: 2.5
Kernel Version: 5.14
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Infiniband/RDMA
Assignee: [email protected]
Reporter: [email protected]
Regression: No

RoCE RDMA connection uses CMA protocol to establish an RDMA connection. During the setup the code uses hard coded timeout/retry values. These values are used for when Connect Request is not being answered to to re-try the request. During the re-try attempts the ARP updates of the destination server are ignored. Current timeout values lead to 4+minutes long attempt at connecting to a server that no longer owns the IP since the ARP update happens.

Quoted 4 lines
Quoted 4 lines

The ask is to make the timeout/retry values configurable via procfs or sysfs. This will allow for environments that use RoCE to reduce the timeouts to a more reasonable values and be able to react to the ARP updates faster. Other CMA users (eg IB or others) can continue to use existing values.

I would rather not add a user-facing tunable. The fabric should be better at detecting addressing changes within a reasonable time. It would be helpful to provide a history of why the ARP timeout is so lax -- do certain ULPs rely on it being long?

Quoted 7 lines
Quoted 6 lines

The problem exist in all kernel versions but bugzilla is filed for 5.14 kernel.

The use case is (RoCE-based) NFSoRDMA where a server went down and another server was brought up in its place. RDMA layer introduces 4+ minutes in being able to re-establish an RDMA connection and let IO resume, due to inability to react to the ARP update.

Quoted 7 lines

RDMA-CM has many different timeouts, so I hope that my answer is for the right timeout.

We probably need to extend rdma_connect() to receive remote_cm_response_timeout value, so NFSoRDMA will set it to whatever value its appropriate.

The timewait will be calculated based it in ib_send_cm_req().

I hope a mechanism can be found that behaves the same or nearly the same way for all RDMA fabrics.

For those who are not NFS-savvy:

Simple NFS server failover is typically implemented with a heartbeat between two similar platforms that both access the same backend storage. When one platform fails, the other detects it and takes over the failing platform's IP address. Clients detect connection loss with the failing platform, and upon reconnection to that IP address are transparently directed to the other platform.

NFS server vendors have tried to extend this behavior to RDMA fabrics, with varying degrees of success.

In addition to enforcing availability SLAs, the time it takes to re-establish a working connection is critical for NFSv4 because each client maintains a lease to prevent the server from purging open and lock state. If the reconnect takes too long, the client's lease is jeopardized because other clients can then access files that client might still have locked or open.

--
Chuck Lever

#3 lore raw
Quoted 38 lines · Chuck Lever III

Hi Leon-

Thanks for the suggestion! More below.

Quoted 29 lines · Leon Romanovsky
Quoted 26 lines · On Fri, Sep 24, 2021 at 03:34:32PM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=214523

Bug ID: 214523
Summary: RDMA Mellanox RoCE drivers are unresponsive to ARP
updates during a reconnect
Product: Drivers
Version: 2.5
Kernel Version: 5.14
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Infiniband/RDMA
Assignee: [email protected]
Reporter: [email protected]
Regression: No

RoCE RDMA connection uses CMA protocol to establish an RDMA connection. During the setup the code uses hard coded timeout/retry values. These values are used for when Connect Request is not being answered to to re-try the request. During the re-try attempts the ARP updates of the destination server are ignored. Current timeout values lead to 4+minutes long attempt at connecting to a server that no longer owns the IP since the ARP update happens.

Quoted 4 lines
Quoted 4 lines

The ask is to make the timeout/retry values configurable via procfs or sysfs. This will allow for environments that use RoCE to reduce the timeouts to a more reasonable values and be able to react to the ARP updates faster. Other CMA users (eg IB or others) can continue to use existing values.

Quoted 4 lines

I would rather not add a user-facing tunable. The fabric should be better at detecting addressing changes within a reasonable time. It would be helpful to provide a history of why the ARP timeout is so lax -- do certain ULPs rely on it being long?

I don't know about ULPs and ARPs, but how to calculate TimeWait is described in the spec.

Regarding tunable, I agree. Because it needs to be per-connection, most likely not many people in the world will success to configure it properly.

Quoted 12 lines
Quoted 12 lines
Quoted 8 lines

The problem exist in all kernel versions but bugzilla is filed for 5.14 kernel.

The use case is (RoCE-based) NFSoRDMA where a server went down and another server was brought up in its place. RDMA layer introduces 4+ minutes in being able to re-establish an RDMA connection and let IO resume, due to inability to react to the ARP update.

RDMA-CM has many different timeouts, so I hope that my answer is for the right timeout.

Quoted 7 lines
Quoted 4 lines

We probably need to extend rdma_connect() to receive remote_cm_response_timeout value, so NFSoRDMA will set it to whatever value its appropriate.

The timewait will be calculated based it in ib_send_cm_req().

I hope a mechanism can be found that behaves the same or nearly the same way for all RDMA fabrics.

It depends on the fabric itself, in every network
remote_cm_response_timeout can be different.

Quoted 26 lines

For those who are not NFS-savvy:

Simple NFS server failover is typically implemented with a heartbeat between two similar platforms that both access the same backend storage. When one platform fails, the other detects it and takes over the failing platform's IP address. Clients detect connection loss with the failing platform, and upon reconnection to that IP address are transparently directed to the other platform.

NFS server vendors have tried to extend this behavior to RDMA fabrics, with varying degrees of success.

In addition to enforcing availability SLAs, the time it takes to re-establish a working connection is critical for NFSv4 because each client maintains a lease to prevent the server from purging open and lock state. If the reconnect takes too long, the client's lease is jeopardized because other clients can then access files that client might still have locked or open.

--
Chuck Lever

#4 lore raw
Quoted 44 lines · Leon Romanovsky
Quoted 38 lines · Chuck Lever III

Hi Leon-

Thanks for the suggestion! More below.

Quoted 29 lines · Leon Romanovsky
Quoted 26 lines · On Fri, Sep 24, 2021 at 03:34:32PM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=214523

Bug ID: 214523
Summary: RDMA Mellanox RoCE drivers are unresponsive to ARP
updates during a reconnect
Product: Drivers
Version: 2.5
Kernel Version: 5.14
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Infiniband/RDMA
Assignee: [email protected]
Reporter: [email protected]
Regression: No

RoCE RDMA connection uses CMA protocol to establish an RDMA connection. During the setup the code uses hard coded timeout/retry values. These values are used for when Connect Request is not being answered to to re-try the request. During the re-try attempts the ARP updates of the destination server are ignored. Current timeout values lead to 4+minutes long attempt at connecting to a server that no longer owns the IP since the ARP update happens.

Quoted 4 lines
Quoted 4 lines

The ask is to make the timeout/retry values configurable via procfs or sysfs. This will allow for environments that use RoCE to reduce the timeouts to a more reasonable values and be able to react to the ARP updates faster. Other CMA users (eg IB or others) can continue to use existing values.

Quoted 4 lines

I would rather not add a user-facing tunable. The fabric should be better at detecting addressing changes within a reasonable time. It would be helpful to provide a history of why the ARP timeout is so lax -- do certain ULPs rely on it being long?

Quoted 5 lines

I don't know about ULPs and ARPs, but how to calculate TimeWait is described in the spec.

Regarding tunable, I agree. Because it needs to be per-connection, most likely not many people in the world will success to configure it properly.

Maybe we should be disconnecting the cm_id if a gratituous ARP changes the MAC address? The cm_id is surely broken after that event right?

Jason

#5 lore raw
Quoted 48 lines · Jason Gunthorpe

External email: Use caution opening links or attachments

Quoted 44 lines · Leon Romanovsky
Quoted 42 lines · Chuck Lever III

Hi Leon-

Thanks for the suggestion! More below.

Quoted 29 lines · Leon Romanovsky
Quoted 26 lines · On Fri, Sep 24, 2021 at 03:34:32PM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=214523

Bug ID: 214523
Summary: RDMA Mellanox RoCE drivers are unresponsive to ARP
updates during a reconnect
Product: Drivers
Version: 2.5
Kernel Version: 5.14
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Infiniband/RDMA
Assignee: [email protected]
Reporter: [email protected]
Regression: No

RoCE RDMA connection uses CMA protocol to establish an RDMA connection. During the setup the code uses hard coded timeout/retry values. These values are used for when Connect Request is not being answered to to re-try the request. During the re-try attempts the ARP updates of the destination server are ignored. Current timeout values lead to 4+minutes long attempt at connecting to a server that no longer owns the IP since the ARP update happens.

Quoted 4 lines
Quoted 4 lines

The ask is to make the timeout/retry values configurable via procfs or sysfs. This will allow for environments that use RoCE to reduce the timeouts to a more reasonable values and be able to react to the ARP updates faster. Other CMA users (eg IB or others) can continue to use existing values.

I would rather not add a user-facing tunable. The fabric should be better at detecting addressing changes within a reasonable time. It would be helpful to provide a history of why the ARP timeout is so lax -- do certain ULPs rely on it being long?

Quoted 8 lines
Quoted 5 lines

I don't know about ULPs and ARPs, but how to calculate TimeWait is described in the spec.

Regarding tunable, I agree. Because it needs to be per-connection, most likely not many people in the world will success to configure it properly.

Maybe we should be disconnecting the cm_id if a gratituous ARP changes the MAC address? The cm_id is surely broken after that event right?

Is there an event on gratuitous ARP? And we also need to notify user-space application, right?

#6 lore raw
Quoted 52 lines · Mark Zhang
Quoted 51 lines · Jason Gunthorpe

External email: Use caution opening links or attachments

Quoted 47 lines · Leon Romanovsky
Quoted 38 lines · Chuck Lever III

Hi Leon-

Thanks for the suggestion! More below.

Quoted 29 lines · Leon Romanovsky
Quoted 26 lines · On Fri, Sep 24, 2021 at 03:34:32PM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=214523

Bug ID: 214523
Summary: RDMA Mellanox RoCE drivers are unresponsive to ARP
updates during a reconnect
Product: Drivers
Version: 2.5
Kernel Version: 5.14
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Infiniband/RDMA
Assignee: [email protected]
Reporter: [email protected]
Regression: No

RoCE RDMA connection uses CMA protocol to establish an RDMA connection. During the setup the code uses hard coded timeout/retry values. These values are used for when Connect Request is not being answered to to re-try the request. During the re-try attempts the ARP updates of the destination server are ignored. Current timeout values lead to 4+minutes long attempt at connecting to a server that no longer owns the IP since the ARP update happens.

Quoted 4 lines
Quoted 4 lines

The ask is to make the timeout/retry values configurable via procfs or sysfs. This will allow for environments that use RoCE to reduce the timeouts to a more reasonable values and be able to react to the ARP updates faster. Other CMA users (eg IB or others) can continue to use existing values.

Quoted 4 lines

I would rather not add a user-facing tunable. The fabric should be better at detecting addressing changes within a reasonable time. It would be helpful to provide a history of why the ARP timeout is so lax -- do certain ULPs rely on it being long?

I don't know about ULPs and ARPs, but how to calculate TimeWait is described in the spec.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Regarding tunable, I agree. Because it needs to be per-connection, most likely not many people in the world will success to configure it properly.

Maybe we should be disconnecting the cm_id if a gratituous ARP changes the MAC address? The cm_id is surely broken after that event right?

Is there an event on gratuitous ARP? And we also need to notify user-space application, right?

I think there is a net notifier for this?

Userspace will see it via the CM event we'll need to trigger.

Jason

#7 lore raw
Quoted 60 lines · Jason Gunthorpe
Quoted 55 lines · Mark Zhang
Quoted 54 lines · Jason Gunthorpe

External email: Use caution opening links or attachments

Quoted 49 lines · Leon Romanovsky
Quoted 38 lines · Chuck Lever III

Hi Leon-

Thanks for the suggestion! More below.

Quoted 29 lines · Leon Romanovsky
Quoted 26 lines · On Fri, Sep 24, 2021 at 03:34:32PM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=214523

Bug ID: 214523
Summary: RDMA Mellanox RoCE drivers are unresponsive to ARP
updates during a reconnect
Product: Drivers
Version: 2.5
Kernel Version: 5.14
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Infiniband/RDMA
Assignee: [email protected]
Reporter: [email protected]
Regression: No

RoCE RDMA connection uses CMA protocol to establish an RDMA connection. During the setup the code uses hard coded timeout/retry values. These values are used for when Connect Request is not being answered to to re-try the request. During the re-try attempts the ARP updates of the destination server are ignored. Current timeout values lead to 4+minutes long attempt at connecting to a server that no longer owns the IP since the ARP update happens.

Quoted 4 lines
Quoted 4 lines

The ask is to make the timeout/retry values configurable via procfs or sysfs. This will allow for environments that use RoCE to reduce the timeouts to a more reasonable values and be able to react to the ARP updates faster. Other CMA users (eg IB or others) can continue to use existing values.

Quoted 4 lines

I would rather not add a user-facing tunable. The fabric should be better at detecting addressing changes within a reasonable time. It would be helpful to provide a history of why the ARP timeout is so lax -- do certain ULPs rely on it being long?

I don't know about ULPs and ARPs, but how to calculate TimeWait is described in the spec.

Regarding tunable, I agree. Because it needs to be per-connection, most likely not many people in the world will success to configure it properly.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Maybe we should be disconnecting the cm_id if a gratituous ARP changes the MAC address? The cm_id is surely broken after that event right?

Is there an event on gratuitous ARP? And we also need to notify user-space application, right?

I think there is a net notifier for this?

NETEVENT_NEIGH_UPDATE may be?

Thxs, Håkon

Quoted 4 lines

Userspace will see it via the CM event we'll need to trigger.

Jason

#8 lore raw
Quoted 64 lines · Haakon Bugge

External email: Use caution opening links or attachments

Quoted 61 lines · Jason Gunthorpe
Quoted 58 lines · Mark Zhang
Quoted 56 lines · Jason Gunthorpe

External email: Use caution opening links or attachments

Quoted 44 lines · Leon Romanovsky
Quoted 38 lines · Chuck Lever III

Hi Leon-

Thanks for the suggestion! More below.

Quoted 29 lines · Leon Romanovsky
Quoted 26 lines · On Fri, Sep 24, 2021 at 03:34:32PM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=214523

Bug ID: 214523
Summary: RDMA Mellanox RoCE drivers are unresponsive to ARP
updates during a reconnect
Product: Drivers
Version: 2.5
Kernel Version: 5.14
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Infiniband/RDMA
Assignee: [email protected]
Reporter: [email protected]
Regression: No

RoCE RDMA connection uses CMA protocol to establish an RDMA connection. During the setup the code uses hard coded timeout/retry values. These values are used for when Connect Request is not being answered to to re-try the request. During the re-try attempts the ARP updates of the destination server are ignored. Current timeout values lead to 4+minutes long attempt at connecting to a server that no longer owns the IP since the ARP update happens.

Quoted 4 lines
Quoted 4 lines

The ask is to make the timeout/retry values configurable via procfs or sysfs. This will allow for environments that use RoCE to reduce the timeouts to a more reasonable values and be able to react to the ARP updates faster. Other CMA users (eg IB or others) can continue to use existing values.

Quoted 4 lines

I would rather not add a user-facing tunable. The fabric should be better at detecting addressing changes within a reasonable time. It would be helpful to provide a history of why the ARP timeout is so lax -- do certain ULPs rely on it being long?

Quoted 5 lines

I don't know about ULPs and ARPs, but how to calculate TimeWait is described in the spec.

Regarding tunable, I agree. Because it needs to be per-connection, most likely not many people in the world will success to configure it properly.

Maybe we should be disconnecting the cm_id if a gratituous ARP changes the MAC address? The cm_id is surely broken after that event right?

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Is there an event on gratuitous ARP? And we also need to notify user-space application, right?

I think there is a net notifier for this?

NETEVENT_NEIGH_UPDATE may be?

How about do it like this:

1. In cma.c we do register_netevent_notifier();
2. On each NETEVENT_NEIGH_UPDATE event, in netevent_callback():
2.1. Allocate a work (as seems the cb is in interrupt context); 2.2. In the new work:

          foreach(cm_dev) {
              foreach(id_priv) {
                  if ((id_priv.dst_ip == event.ip) &&
                      (id_priv.dst_addr != event.ha)) {

                      /* Anything more to do? */
                      report_event(RDMA_CM_EVENT_ADDR_CHANGE);
                  }
              }
          }

And I have these questions:
1. Should we do it in cma.c or cm.c?
2. Should we do register only once, or per id? If we do register per id then there maybe many ids;
3. If we do it in cm.c, then should we do more like ib_cancel_mad()? Or report an event is enough?
4. Need to create a work on each ARP event, would it be a heavy load?
5. Do we need a new event, instead of RDMA_CM_EVENT_ADDR_CHANGE?
6. How about if peer is not in same subnet?

Thank you very much.

#9 lore raw
In reply to Leon Romanovsky
Quoted 48 lines · Leon Romanovsky
Quoted 38 lines · Chuck Lever III

Hi Leon-

Thanks for the suggestion! More below.

Quoted 29 lines · Leon Romanovsky
Quoted 26 lines · On Fri, Sep 24, 2021 at 03:34:32PM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=214523

Bug ID: 214523
Summary: RDMA Mellanox RoCE drivers are unresponsive to ARP
updates during a reconnect
Product: Drivers
Version: 2.5
Kernel Version: 5.14
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Infiniband/RDMA
Assignee: [email protected]
Reporter: [email protected]
Regression: No

RoCE RDMA connection uses CMA protocol to establish an RDMA connection. During the setup the code uses hard coded timeout/retry values. These values are used for when Connect Request is not being answered to to re-try the request. During the re-try attempts the ARP updates of the destination server are ignored. Current timeout values lead to 4+minutes long attempt at connecting to a server that no longer owns the IP since the ARP update happens.

Quoted 4 lines
Quoted 4 lines

The ask is to make the timeout/retry values configurable via procfs or sysfs. This will allow for environments that use RoCE to reduce the timeouts to a more reasonable values and be able to react to the ARP updates faster. Other CMA users (eg IB or others) can continue to use existing values.

Quoted 4 lines

I would rather not add a user-facing tunable. The fabric should be better at detecting addressing changes within a reasonable time. It would be helpful to provide a history of why the ARP timeout is so lax -- do certain ULPs rely on it being long?

Quoted 5 lines

I don't know about ULPs and ARPs, but how to calculate TimeWait is described in the spec.

Regarding tunable, I agree. Because it needs to be per-connection, most likely not many people in the world will success to configure it properly.

Exactly.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 6 lines

The problem exist in all kernel versions but bugzilla is filed for 5.14 kernel.

The use case is (RoCE-based) NFSoRDMA where a server went down and another server was brought up in its place. RDMA layer introduces 4+ minutes in being able to re-establish an RDMA connection and let IO resume, due to inability to react to the ARP update.

RDMA-CM has many different timeouts, so I hope that my answer is for the right timeout.

We probably need to extend rdma_connect() to receive remote_cm_response_timeout value, so NFSoRDMA will set it to whatever value its appropriate.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

The timewait will be calculated based it in ib_send_cm_req().

I hope a mechanism can be found that behaves the same or nearly the same way for all RDMA fabrics.

It depends on the fabric itself, in every network
remote_cm_response_timeout can be different.

What I mean is I hope a way can be found so that RDMA consumers do not have to be aware of the fabric differences.

Quoted 20 lines
Quoted 20 lines

For those who are not NFS-savvy:

Simple NFS server failover is typically implemented with a heartbeat between two similar platforms that both access the same backend storage. When one platform fails, the other detects it and takes over the failing platform's IP address. Clients detect connection loss with the failing platform, and upon reconnection to that IP address are transparently directed to the other platform.

NFS server vendors have tried to extend this behavior to RDMA fabrics, with varying degrees of success.

In addition to enforcing availability SLAs, the time it takes to re-establish a working connection is critical for NFSv4 because each client maintains a lease to prevent the server from purging open and lock state. If the reconnect takes too long, the client's lease is jeopardized because other clients can then access files that client might still have locked or open.

Quoted 2 lines
Quoted 2 lines

--
Chuck Lever

--
Chuck Lever