can: m_can: Control tx flow to avoid message stuck
Subsystems (4)
PATCH 1/1 can: m_can: Control tx flow to avoid message stuck
From: Subramanian Mohan <[email protected]>
The prolonged testing of passing can messages between two Elkhartlake platforms resulted in message stuck i.e Message did not receive at receiver side
Contolling TX i.e TEFN bit helped to resolve the message stuck issue.
The current solution is enhanced/optimized from the below patch:
https://lore.kernel.org/lkml/[email protected]/T/
Setup used to reproduce the issue:
+---------------------+ +----------------------+ |Intel ElkhartLake | |Intel ElkhartLake | | +--------+ | | +--------+ | | |m_can 0 | |<=======>| |m_can 0 | | | +--------+ | | +--------+ | +---------------------+ +----------------------+
Steps to be run on the two Elkhartlake HW:
1)Bus-Rate is 1 MBit/s
2)Busload during the test is about 40%
3)we initialize the CAN with following commands 4)ip link set can0 txqueuelen 100/1024/2048
5)ip link set can0 up type can bitrate 1000000
Python scripts are used send and receive the can messages between the EHL systems.
--- drivers/net/can/m_can/m_can.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/net/can/m_can/m_can.c +9 -2
@@ -1220,7 +1220,7 @@ static void m_can_coalescing_update(struct m_can_classdev *cdev, u32 ir)static int m_can_interrupt_handler (struct m_can_classdev *cdev) {struct net_device *dev = cdev->net; -u32 ir =0 , ir_read; +u32 ir =0 , ir_read, new_interrupts;int ret;if (pm_runtime_suspended (cdev->dev))
@@ -1283,6 +1283,9 @@ static int m_can_interrupt_handler(struct m_can_classdev *cdev) ret =m_can_echo_tx_event (dev);if (ret !=0 )return ret; + + new_interrupts = cdev->active_interrupts & ~(IR_TEFN ); +m_can_interrupt_enable (cdev, new_interrupts); } }
@@ -1989,6 +1992,7 @@ static netdev_tx_t m_can_start_xmit(struct sk_buff *skb,struct m_can_classdev *cdev =netdev_priv (dev);unsigned int frame_len;netdev_tx_t ret; +u32 new_interrupts;if (can_dev_dropped_skb (dev, skb))return NETDEV_TX_OK ;
@@ -2008,8 +2012,11 @@ static netdev_tx_t m_can_start_xmit(struct sk_buff *skb,if (cdev->is_peripheral) ret =m_can_start_peripheral_xmit (cdev, skb); -else +else { + new_interrupts = cdev->active_interrupts |IR_TEFN ; +m_can_interrupt_enable (cdev, new_interrupts); ret =m_can_tx_handler (cdev, skb); + }if (ret !=NETDEV_TX_OK )netdev_completed_queue (dev,1 , frame_len);
Signature · 1 line
2.35.3
Markus Schneider-Pargmann
maintainerHi,
Quoted 5 lines · On Wed, Jan 08, 2025 at 02:31:12PM +0530, [email protected] wrote:
From: Subramanian Mohan <[email protected]>
The prolonged testing of passing can messages between two Elkhartlake platforms resulted in message stuck i.e Message did not receive at receiver side
Can you please describe the reason for the stuck messages in your commit message? I am reading this but I don't understand why this happens or why your proposed solution helps.
Quoted 46 lines
Contolling TX i.e TEFN bit helped to resolve the message stuck issue.
The current solution is enhanced/optimized from the below patch:
https://lore.kernel.org/lkml/[email protected]/T/Setup used to reproduce the issue:
+---------------------+ +----------------------+ |Intel ElkhartLake | |Intel ElkhartLake | | +--------+ | | +--------+ | | |m_can 0 | |<=======>| |m_can 0 | | | +--------+ | | +--------+ | +---------------------+ +----------------------+Steps to be run on the two Elkhartlake HW:
1)Bus-Rate is 1 MBit/s
2)Busload during the test is about 40%
3)we initialize the CAN with following commands 4)ip link set can0 txqueuelen 100/1024/2048
5)ip link set can0 up type can bitrate 1000000Python scripts are used send and receive the can messages between the EHL systems.
Signed-off-by Hahn Matthias <[email protected]>Signed-off-by Subramanian Mohan <[email protected]>--- drivers/net/can/m_can/m_can.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)drivers/net/can/m_can/m_can.c +1
-1@@ -1220,7 +1220,7 @@ static void m_can_coalescing_update(struct m_can_classdev *cdev, u32 ir)static int m_can_interrupt_handler (struct m_can_classdev *cdev) {struct net_device *dev = cdev->net; -u32 ir =0 , ir_read; +u32 ir =0 , ir_read, new_interrupts;int ret;if (pm_runtime_suspended (cdev->dev))
drivers/net/can/m_can/m_can.c static int m_can_interrupt_handler(struct m_can_classdev *cdev) · line 1288
Quoted 6 lines
drivers/net/can/m_can/m_can.c +3
-0@@ -1283,6 +1283,9 @@ static int m_can_interrupt_handler(struct m_can_classdev *cdev) ret =m_can_echo_tx_event (dev);if (ret !=0 )return ret; + + new_interrupts = cdev->active_interrupts & ~(IR_TEFN ); +m_can_interrupt_enable (cdev, new_interrupts);
Here is a theoretical situation of two messages being sent. The first is being sent and handled in this interrupt handler. Then it would disable the TEFN bit right? If the second message wasn't done sending yet, how would it ever call the interrupt handler if the interrupt is disabled?
Also you are disabling this interrupt here regardless of the type of mcan device and also regardless of the coalescing state. In the transmit part you are only enabling it for non-peripheral devices. For peripheral mcan devices this would also introduce an additional two transfers per transmit.
In which situations is this really necessary? Does it help to implement coalescing for non-peripheral devices?
Best
Markus
Quoted 27 lines
drivers/net/can/m_can/m_can.c +5
-1} }@@ -1989,6 +1992,7 @@ static netdev_tx_t m_can_start_xmit(struct sk_buff *skb,struct m_can_classdev *cdev =netdev_priv (dev);unsigned int frame_len;netdev_tx_t ret; +u32 new_interrupts;if (can_dev_dropped_skb (dev, skb))return NETDEV_TX_OK ;@@ -2008,8 +2012,11 @@ static netdev_tx_t m_can_start_xmit(struct sk_buff *skb,if (cdev->is_peripheral) ret =m_can_start_peripheral_xmit (cdev, skb); -else +else { + new_interrupts = cdev->active_interrupts |IR_TEFN ; +m_can_interrupt_enable (cdev, new_interrupts); ret =m_can_tx_handler (cdev, skb); + }if (ret !=NETDEV_TX_OK )netdev_completed_queue (dev,1 , frame_len);Signature · 1 line
2.35.3
A message that is not in the archive (sent off-list or not yet received): PH7PR11MB58625A0D029BB135A9F506C8F71F2@PH7PR11MB5862.namprd11.prod.outlook.com
Hi @Markus Schneider-Pargmann,
Quoted 19 lines
-----Original Message-----
From: Markus Schneider-Pargmann <mailto:[email protected]>
Sent: Thursday, January 9, 2025 9:14 PM
To: Mohan, Subramanian <mailto:[email protected]>
Cc: mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; Tan, Raymond <mailto:[email protected]>; mailto:[email protected]; linux- mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; mailto:[email protected]; Hahn, Matthias <mailto:[email protected]>; Chinnadurai, Srinivasan <mailto:[email protected]>
Subject: Re: [PATCH 1/1] can:m_can: Control tx flow to avoid message stuckHi,
Quoted 2 lines · On Wed, Jan 08, 2025 at 02:31:12PM +0530, mailto:[email protected]…
From: Subramanian Mohan <mailto:[email protected]>
Quoted 7 lines
Quoted 3 lines
The prolonged testing of passing can messages between two Elkhartlake platforms resulted in message stuck i.e Message did not receive at receiver side
Can you please describe the reason for the stuck messages in your commit message? I am reading this but I don't understand why this happens or why your proposed solution helps.
Let me describe problem bit more:
We are using 2 different Python Scripts(client and server) on both of the Elkhart lake connected systems.
The "server" script sends out messages with Arbitration ID's, and then waits for a response. If the Arbitration ID is different than the one expected or no message arrives it logs an error.
The "client" script listens for messages, and depending on the Arbitration ID received it sends a message with a specific Arbitration ID back. We have deployed both the scripts in 2 different systems and triggered the testing
If any message is lost/stuck then the "server" - Script will log an error.
The Message stuck corresponds over here, whenever the server sends out message and waits for reply, we wont me getting the reply message On the server side. Even though time slice increase in scripts did not help. On further debugging enabling TX/TEFN impacts the processing load. To overcome this we disabled the TX/TEFN interrupt once processed and enable it back in the TX start xmit function.
Quoted 53 lines
Quoted 45 lines
Contolling TX i.e TEFN bit helped to resolve the message stuck issue.
The current solution is enhanced/optimized from the below patch:
https://lore.kernel.org/lkml/20230623051124.64132-1-kumari.pallavi@int
el.com/T/Setup used to reproduce the issue:
+---------------------+ +----------------------+ |Intel ElkhartLake | |Intel ElkhartLake | | +--------+ | | +--------+ | | |m_can 0 | |<=======>| |m_can 0 | | | +--------+ | | +--------+ | +---------------------+ +----------------------+Steps to be run on the two Elkhartlake HW:
1)Bus-Rate is 1 MBit/s
2)Busload during the test is about 40% 3)we initialize the CAN with following commands 4)ip link set can0 txqueuelen 100/1024/2048 5)ip link set can0 up type can bitrate 1000000Python scripts are used send and receive the can messages between the EHL systems.
Signed-off-by Hahn Matthias <mailto:[email protected]>Signed-off-by Subramanian Mohan <mailto:[email protected]>---
drivers/net/can/m_can/m_can.c| 11 +++++++++--1 file changed, 9 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/drivers/net/can/m_can/m_can.cindex97cd8bbf2e32..0a2c9a622842100644drivers/net/can/m_can/m_can.c +0
-0@@ -1220,7 +1220,7 @@ static void m_can_coalescing_update(struct
m_can_classdev*cdev, u32 ir) static intm_can_interrupt_handler(struct m_can_classdev *cdev){ structnet_device*dev =cdev->net;
- u32 ir = 0,ir_read;
+ u32 ir = 0,ir_read,new_interrupts;
int ret;Quoted 1 line
if (
pm_runtime_suspended(cdev->dev)) @@ -1283,6 +1283,9 @@static
Quoted 6 lines
int
m_can_interrupt_handler(struct m_can_classdev *cdev)ret =m_can_echo_tx_event(dev); if (ret != 0)
return ret;
+
+new_interrupts=cdev->active_interrupts&
Quoted 7 lines
~(IR_TEFN);
Quoted 1 line
+
m_can_interrupt_enable(cdev, new_interrupts);Here is a theoretical situation of two messages being sent. The first is being sent and handled in this interrupt handler. Then it would disable the TEFN bit right? If the second message wasn't done sending yet, how would it ever call the interrupt handler if the interrupt is disabled?
With this patch we are controlling only TEFN/TX interrupt bit, rest of the interrupts remains unaffected. Since We are enabling/disabling TEFN bit only, interrupt handler will be called normally with other interrupts.
Quoted 5 lines
Also you are disabling this interrupt here regardless of the type of mcan device and also regardless of the coalescing state. In the transmit part you are only enabling it for non-peripheral devices. For peripheral mcan devices this would also introduce an additional two transfers per transmit.
TEFN bit enabling/disabling applies only to non-peripheral device.
While disabling the TEFN bit in interrupt handler, we will add the check for non-peripheral device before disabling it(V2). On the coalescing state, The snapshot is already taken while entering the interrupt handler.
Quoted 3 lines
In which situations is this really necessary? Does it help to implement coalescing for non-peripheral devices?
This helps in heavy load/traffic conditions
Not exactly sure on the coalescing part.
Thanks,
Subbu
Quoted 33 lines
Best
Markusdrivers/net/can/m_can/m_can.c static netdev_tx_t m_can_start_xmit(struct
Quoted 4 lines
}
}drivers/net/can/m_can/m_can.c +0
-0@@ -1989,6 +1992,7 @@ static netdev_tx_t m_can_start_xmit(struct
sk_buff*skb,Quoted 24 lines
drivers/net/can/m_can/m_can.c +5
-1struct m_can_classdev *cdev = netdev_priv(dev); unsigned int frame_len; netdev_tx_t ret; + u32 new_interrupts; if (can_dev_dropped_skb(dev, skb)) return NETDEV_TX_OK;@@ -2008,8 +2012,11 @@ static netdev_tx_t m_can_start_xmit(struct sk_buff *skb, if (cdev->is_peripheral) ret = m_can_start_peripheral_xmit(cdev, skb); - else + else { + new_interrupts = cdev->active_interrupts | IR_TEFN; + m_can_interrupt_enable(cdev, new_interrupts); ret = m_can_tx_handler(cdev, skb); + } if (ret != NETDEV_TX_OK)
netdev_completed_queue(dev, 1, frame_len); --
2.35.3
Markus Schneider-Pargmann
maintainerQuoted 29 lines · Subramanian
Hi @Markus Schneider-Pargmann,
Quoted 19 lines
-----Original Message-----
From: Markus Schneider-Pargmann <[email protected]>
Sent: Thursday, January 9, 2025 9:14 PM
To: Mohan, Subramanian <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Tan, Raymond <[email protected]>; [email protected]; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Hahn, Matthias <[email protected]>; Chinnadurai, Srinivasan <[email protected]>
Subject: Re: [PATCH 1/1] can:m_can: Control tx flow to avoid message stuckHi,
Quoted 2 lines · On Wed, Jan 08, 2025 at 02:31:12PM +0530, [email protected] wrote:
From: Subramanian Mohan <[email protected]>
Quoted 7 lines
Quoted 3 lines
The prolonged testing of passing can messages between two Elkhartlake platforms resulted in message stuck i.e Message did not receive at receiver side
Can you please describe the reason for the stuck messages in your commit message? I am reading this but I don't understand why this happens or why your proposed solution helps.
Quoted 10 lines
Let me describe problem bit more:
We are using 2 different Python Scripts(client and server) on both of the Elkhart lake connected systems.
The "server" script sends out messages with Arbitration ID's, and then waits for a response. If the Arbitration ID is different than the one expected or no message arrives it logs an error.
The "client" script listens for messages, and depending on the Arbitration ID received it sends a message with a specific Arbitration ID back. We have deployed both the scripts in 2 different systems and triggered the testing
If any message is lost/stuck then the "server" - Script will log an error.
The Message stuck corresponds over here, whenever the server sends out message and waits for reply, we wont me getting the reply message On the server side. Even though time slice increase in scripts did not help. On further debugging enabling TX/TEFN impacts the processing load. To overcome this we disabled the TX/TEFN interrupt once processed and enable it back in the TX start xmit function.
Sorry, I still don't understand what is happening in the driver/hardware that requires the interrupt to be disabled and enabled again later. Is it causing interrupt_handler runs that are not associated with a real interrupt? Is it causing real interrupts without anything being sent? Does the clearing of the interrupts not work?
Quoted 57 lines
Quoted 57 lines
Quoted 42 lines
Contolling TX i.e TEFN bit helped to resolve the message stuck issue.
The current solution is enhanced/optimized from the below patch:
https://lore.kernel.org/lkml/20230623051124.64132-1-kumari.pallavi@int
el.com/T/Setup used to reproduce the issue:
+---------------------+ +----------------------+ |Intel ElkhartLake | |Intel ElkhartLake | | +--------+ | | +--------+ | | |m_can 0 | |<=======>| |m_can 0 | | | +--------+ | | +--------+ | +---------------------+ +----------------------+Steps to be run on the two Elkhartlake HW:
1)Bus-Rate is 1 MBit/s
2)Busload during the test is about 40% 3)we initialize the CAN with following commands 4)ip link set can0 txqueuelen 100/1024/2048 5)ip link set can0 up type can bitrate 1000000Python scripts are used send and receive the can messages between the EHL systems.
Signed-off-by Hahn Matthias <[email protected]>Signed-off-by Subramanian Mohan <[email protected]>--- drivers/net/can/m_can/m_can.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/drivers/net/can/m_can/m_can.cindex97cd8bbf2e32..0a2c9a622842100644drivers/net/can/m_can/m_can.c +0
-0@@ -1220,7 +1220,7 @@ static void m_can_coalescing_update(struct
m_can_classdev*cdev, u32 ir) static intm_can_interrupt_handler(struct m_can_classdev *cdev) { struct net_device *dev = cdev->net;drivers/net/can/m_can/m_can.c static void m_can_coalescing_update(struct m_can_classdev *cdev, u32 ir)
Quoted 5 lines
drivers/net/can/m_can/m_can.c +1
-1-u32 ir =0 , ir_read; +u32 ir =0 , ir_read, new_interrupts;int ret; if (pm_runtime_suspended(cdev->dev)) @@ -1283,6 +1283,9 @@static
drivers/net/can/m_can/m_can.c
Quoted 6 lines
int
m_can_interrupt_handler(struct m_can_classdev *cdev)ret = m_can_echo_tx_event(dev); if (ret != 0) return ret;drivers/net/can/m_can/m_can.c +2
-0+ + new_interrupts = cdev->active_interrupts &~(IR_TEFN);
Quoted 1 line
+
m_can_interrupt_enable(cdev, new_interrupts);
Quoted 7 lines
Quoted 4 lines
Here is a theoretical situation of two messages being sent. The first is being sent and handled in this interrupt handler. Then it would disable the TEFN bit right? If the second message wasn't done sending yet, how would it ever call the interrupt handler if the interrupt is disabled?
With this patch we are controlling only TEFN/TX interrupt bit, rest of the interrupts remains unaffected. Since We are enabling/disabling TEFN bit only, interrupt handler will be called normally with other interrupts.
I am worried about the next TEFN interrupt here. I am currently not sure, can the mcan driver for non-peripheral setups have multiple messages being sent at the same time? If that is possible then what happens to the other messages that are in flight when you disable TEFN?
Best
Markus
Quoted 53 lines
Quoted 6 lines
Also you are disabling this interrupt here regardless of the type of mcan device and also regardless of the coalescing state. In the transmit part you are only enabling it for non-peripheral devices. For peripheral mcan devices this would also introduce an additional two transfers per transmit.
TEFN bit enabling/disabling applies only to non-peripheral device.
While disabling the TEFN bit in interrupt handler, we will add the check for non-peripheral device before disabling it(V2). On the coalescing state, The snapshot is already taken while entering the interrupt handler.Quoted 3 lines
In which situations is this really necessary? Does it help to implement coalescing for non-peripheral devices?
This helps in heavy load/traffic conditions
Not exactly sure on the coalescing part.Thanks,
SubbuQuoted 33 lines
Best
Markusdrivers/net/can/m_can/m_can.c static netdev_tx_t m_can_start_xmit(struct
Quoted 4 lines
} }drivers/net/can/m_can/m_can.c +0
-0@@ -1989,6 +1992,7 @@ static netdev_tx_t m_can_start_xmit(struct
sk_buff*skb,Quoted 24 lines
drivers/net/can/m_can/m_can.c +5
-1struct m_can_classdev *cdev =netdev_priv (dev);unsigned int frame_len;netdev_tx_t ret; +u32 new_interrupts;if (can_dev_dropped_skb (dev, skb))return NETDEV_TX_OK ;@@ -2008,8 +2012,11 @@ static netdev_tx_t m_can_start_xmit(struct sk_buff *skb, if (cdev->is_peripheral) ret = m_can_start_peripheral_xmit(cdev, skb); - else + else { + new_interrupts = cdev->active_interrupts |IR_TEFN ; +m_can_interrupt_enable (cdev, new_interrupts); ret =m_can_tx_handler (cdev, skb); + }if (ret !=NETDEV_TX_OK )netdev_completed_queue(dev, 1, frame_len);--
2.35.3