Skip to contentSkip to conversation

connectors/cn_pec: improve reliability

PATCH New Bogdan Lezhepekov 3 participants 6 messages Aug 5, 2021 – Aug 6, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 connectors/cn_pec: improve reliability

Sometimes pec_listener has not enough time to handle all the generated events.

This patch only returns back the delay that was removed in a recent patch.

---
 testcases/kernel/connectors/pec/cn_pec.sh | 2 ++
 1 file changed, 2 insertions(+)
testcases/kernel/connectors/pec/cn_pec.sh +2 -0
index 9b85a5c81..24b1db761 100755
@@ -77,6 +77,8 @@ test()
 	tst_sleep 100ms
 
 	ROD event_generator -n $num_events -e $event \>gen_$event.log 2\>gen_$event.err
+    # sleep until pec_listener has seen and handled all of the generated events
+    tst_sleep 100ms
 
 	kill -s INT $pid 2> /dev/null
 	wait $pid
Signature · 1 line
2.32.0
#1 lore raw

PATCH 1/1 connectors/cn_pec: improve reliability

Hi,

Quoted 5 lines · Bogdan Lezhepekov via ltp

Sometimes pec_listener has not enough time to handle all the generated events.

This patch only returns back the delay that was removed in a recent patch.

I think I thought about keeping or removing this delay and finally decided to remove it for some reason, that I do not remember right now. Can you tell me the environment where this failed?

Joerg

#2 lore raw

PATCH 1/1 connectors/cn_pec: improve reliability

In reply to Bogdan Lezhepekov

Hi Bogdan, Joerg,

Quoted 16 lines

Sometimes pec_listener has not enough time to handle all the generated events.

This patch only returns back the delay that was removed in a recent patch.

---
 testcases/kernel/connectors/pec/cn_pec.sh | 2 ++
 1 file changed, 2 insertions(+)
testcases/kernel/connectors/pec/cn_pec.sh +0 -0
index 9b85a5c81..24b1db761 100755

testcases/kernel/connectors/pec/cn_pec.sh test() · line 81

Quoted 5 lines
testcases/kernel/connectors/pec/cn_pec.sh +2 -0
@@ -77,6 +77,8 @@ test()
 	tst_sleep 100ms
 
 	ROD event_generator -n $num_events -e $event \>gen_$event.log 2\>gen_$event.err
+    # sleep until pec_listener has seen and handled all of the generated events
+    tst_sleep 100ms

nit: mixing tabs and spaces (script uses tabs), can be fixed during merge.

If I understand the code correctly, pec_listener keeps receiving msg from PEC until got signaled by kill. We generally try to avoid sleep in tests [1].

@Joerg: I'm not sure if it's reasonable to add another signal handler in which pec_listener would print number of already handled requests (with write() as printf() is not reentrant). Then we could loop for certain time before sending INT.

Kind regards,
Petr

[1] https://people.kernel.org/metan/why-sleep-is-almost-never-acceptable-in-tests

Quoted 2 lines
testcases/kernel/connectors/pec/cn_pec.sh +0 -0
 	kill -s INT $pid 2> /dev/null
 	wait $pid
#3 lore raw

PATCH 1/1 connectors/cn_pec: improve reliability

Hi Petr,

Quoted 8 lines · Petr Vorel

If I understand the code correctly, pec_listener keeps receiving msg from PEC until got signaled by kill. We generally try to avoid sleep in tests [1].

@Joerg: I'm not sure if it's reasonable to add another signal handler in which pec_listener would print number of already handled requests (with write() as printf() is not reentrant). Then we could loop for certain time before sending INT.

The problem is, that it is hard to know, if all expected events were received. pec_listener receives events from all processes. The only way to reliably handle the issue without sleeping is tracking exit-events in pec_listener and to tell it to die, once it received an exit event from event_generator.
Only then we can be sure, that all events generated by the generator (and hopefully all it's children) were received.

We could also get rid of the other sleep (after starting the listener using some kind of synchronization (I guess right after PROC_CN_MCAST_LISTEN should work). I think checkpoints are not part of the new lib...

Joerg

#4 lore raw

PATCH 1/1 connectors/cn_pec: improve reliability

Hi Joerg, Petr!

Thanks for reviewing and commenting! I don't have a strong opinion on how the test flow should look like, but I can say this sleep definitely makes the test much more reliable.

The results, we get on known to both of you SUSE Carwos, are pretty consistent. My feeling is that 8/10 runs end up with some sporadic failures. The fact that extra sleep solves the problem obviously indicates a lack of synchronization.

@Joerg: if you need more info about the environment, please reach out to me.

-Bogdan
________________________________
From: Joerg Vehlow <[email protected]>
Sent: Friday, August 6, 2021 11:14
To: Petr Vorel <[email protected]>; Bogdan Lezhepekov <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: [LTP] [PATCH v1] connectors/cn_pec: improve reliability

Hi Petr,

Quoted 8 lines · Petr Vorel

If I understand the code correctly, pec_listener keeps receiving msg from PEC until got signaled by kill. We generally try to avoid sleep in tests [1].

@Joerg: I'm not sure if it's reasonable to add another signal handler in which pec_listener would print number of already handled requests (with write() as printf() is not reentrant). Then we could loop for certain time before sending INT.

The problem is, that it is hard to know, if all expected events were received. pec_listener receives events from all processes. The only way to reliably handle the issue without sleeping is tracking exit-events in pec_listener and to tell it to die, once it received an exit event from event_generator.
Only then we can be sure, that all events generated by the generator (and hopefully all it's children) were received.

We could also get rid of the other sleep (after starting the listener using some kind of synchronization (I guess right after PROC_CN_MCAST_LISTEN should work). I think checkpoints are not part of the new lib...

Joerg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20210806/8c6a05fb/attachment-0001.htm>

#5 lore raw

PATCH 1/1 connectors/cn_pec: improve reliability

Hi Bogdan,

Quoted 6 lines · Bogdan Lezhepekov

Hi Joerg, Petr!

Thanks for reviewing and commenting! I don't have a strong opinion on how the test flow should look like, but I can say this sleep definitely makes the test much more reliable.

Quoted 7 lines

The results, we get on known to both of you SUSE Carwos, are pretty consistent. My feeling is that 8/10 runs end up with some sporadic failures. The fact that extra sleep solves the problem obviously indicates a lack of synchronization.

@Joerg: if you need more info about the environment, please reach out to me.

Strange, my test environment was probably something derived from carwos (not exactly sure anymore).
I will try to reproduce and implement a correct synchronization as described in my earlier mail next week.
But maybe you can tell me more details about the actual used environment (If it is related to carwos customer project, you can send it to my corporate email address: [email protected]).

Joerg