infiniband:cma: change iboe packet life time from 18 to 16
PATCH 1/1 infiniband:cma: change iboe packet life time from 18 to 16
The ack timeout retransmission time is affected by the following two
factors: one is packet life time, another is the HCA processing time. Now the default packet lifetime(CMA_IBOE_PACKET_LIFETIME) is 18. That means the minimum ack timeout is 2 seconds(2^(18+1)*4us=2.097seconds). The packet lifetime means the maximum transmission time of packets on the network, 2 seconds is too long.
Assume the network is a clos topology with three layers, every packet will pass through five hops of switches. Assume the buffer of every switch is 128MB and the port transmission rate is 25 Gbit/s, the maximum transmission time of the packet is 200ms(128MB*5/25Gbit/s). Add double redundancy, it is less than 500ms.
So change the CMA_IBOE_PACKET_LIFETIME to 16,
the maximum transmission time of the packet will be about 500+ms, it is long enough.
--- drivers/infiniband/core/cma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/core/cma.c +1 -1
@@ -47,7 +47,7 @@ MODULE_LICENSE("Dual BSD/GPL");#define CMA_CM_RESPONSE_TIMEOUT 20#define CMA_MAX_CM_RETRIES 15#define CMA_CM_MRA_SETTING (IB_CM_MRA_FLAG_DELAY | 24) -#define CMA_IBOE_PACKET_LIFETIME 18 +#define CMA_IBOE_PACKET_LIFETIME 16#define CMA_PREFERRED_ROCE_GID_TYPE IB_GID_TYPE_ROCE_UDP_ENCAPstatic const char *const cma_events[] = {
Signature · 1 line
2.16.4
Hi, jason
Gently ping, thank you.
Quoted 34 lines · Chao Leng
The ack timeout retransmission time is affected by the following two
factors: one is packet life time, another is the HCA processing time. Now the default packetlifetime(CMA_IBOE_PACKET_LIFETIME)is 18. That means the minimum ack timeout is 2 seconds(2^(18+1)*4us=2.097seconds). The packet lifetime means the maximum transmission time of packets on the network, 2 seconds is too long.
Assume the network is a clos topology with three layers, every packet will pass through five hops of switches. Assume the buffer of every switch is 128MB and the port transmission rate is 25 Gbit/s, the maximum transmission time of the packet is 200ms(128MB*5/25Gbit/s). Add double redundancy, it is less than 500ms.
So change theCMA_IBOE_PACKET_LIFETIMEto 16,
the maximum transmission time of the packet will be about 500+ms, it is long enough.Signed-off-by Chao Leng <[email protected]>--- drivers/infiniband/core/cma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/infiniband/core/cma.c +1
-1@@ -47,7 +47,7 @@ MODULE_LICENSE("Dual BSD/GPL");#define CMA_CM_RESPONSE_TIMEOUT 20#define CMA_MAX_CM_RETRIES 15#define CMA_CM_MRA_SETTING (IB_CM_MRA_FLAG_DELAY | 24) -#define CMA_IBOE_PACKET_LIFETIME 18 +#define CMA_IBOE_PACKET_LIFETIME 16#define CMA_PREFERRED_ROCE_GID_TYPE IB_GID_TYPE_ROCE_UDP_ENCAPstatic const char *const cma_events[] = {
Quoted 15 lines · Chao Leng
The ack timeout retransmission time is affected by the following two
factors: one is packet life time, another is the HCA processing time. Now the default packetlifetime(CMA_IBOE_PACKET_LIFETIME)is 18. That means the minimum ack timeout is 2 seconds(2^(18+1)*4us=2.097seconds). The packet lifetime means the maximum transmission time of packets on the network, 2 seconds is too long.
Assume the network is a clos topology with three layers, every packet will pass through five hops of switches. Assume the buffer of every switch is 128MB and the port transmission rate is 25 Gbit/s, the maximum transmission time of the packet is 200ms(128MB*5/25Gbit/s). Add double redundancy, it is less than 500ms.
So change theCMA_IBOE_PACKET_LIFETIMEto 16,
the maximum transmission time of the packet will be about 500+ms, it is long enough.
Quoted 4 lines
Signed-off-by Chao Leng <[email protected]>--- drivers/infiniband/core/cma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Applied to for-next, thanks
Jason
Suri Shelvapille
AppliedIs it possible to make this parameter tunable? In pure IB fabrics I believe you can....
-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, December 7, 2022 6:49 PM
To: Chao Leng <[email protected]>
Cc: [email protected]; [email protected]
Subject: Re: [PATCH] infiniband:cma: change iboe packet life time from 18 to 16
Quoted 14 lines · Chao Leng
The ack timeout retransmission time is affected by the following two
factors: one is packet life time, another is the HCA processing time. Now the default packetlifetime(CMA_IBOE_PACKET_LIFETIME)is 18. That means the minimum ack timeout is 2 seconds(2^(18+1)*4us=2.097seconds). The packet lifetime means the maximum transmission time of packets on the network, 2 seconds is too long.
Assume the network is a clos topology with three layers, every packet will pass through five hops of switches. Assume the buffer of every switch is 128MB and the port transmission rate is 25 Gbit/s, the maximum transmission time of the packet is 200ms(128MB*5/25Gbit/s). Add double redundancy, it is less than 500ms.
So change theCMA_IBOE_PACKET_LIFETIMEto 16, the maximum transmission time of the packet will be about 500+ms, it is long enough.
Quoted 4 lines
Signed-off-by Chao Leng <[email protected]>--- drivers/infiniband/core/cma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Applied to for-next, thanks
Jason
This correspondence, and any attachments or files transmitted with this correspondence, contains information which may be confidential and privileged and is intended solely for the use of the addressee. Unless you are the addressee or are authorized to receive messages for the addressee, you may not use, copy, disseminate, or disclose this correspondence or any information contained in this correspondence to any third party. If you have received this correspondence in error, please notify the sender immediately and delete this correspondence and any attachments or files transmitted with this correspondence from your system, and destroy any and all copies thereof, electronic or otherwise. Your cooperation and understanding are greatly appreciated.