Skip to contentSkip to conversation

perf sched timehist --state vs perf script

DISCUSSION [email protected] 3 participants 4 messages Oct 24, 2024 – Nov 2, 2024

lore
Share & tools
lore
#0 lore raw

From: Veronika Molnarova <[email protected]>

Hello!

We encountered an issue with 'perf sched timehist --state' when we expected for a test case that the process of a workload would eventually end in the dead state(marked by 'X') and be switched out. This can be seen also by using 'perf script':

    # perf sched record -a sleep 1
    [ perf record: Woken up 1 times to write data ]
    [ perf record: Captured and wrote 0.167 MB perf.data (166 samples) ]
    # perf script | grep sleep
           sleep  151541 [000] 193461.410617: sched:sched_stat_runtime: comm=sleep pid=151541 runtime=3132275 [ns]
           sleep  151541 [000] 193461.410624:       sched:sched_switch: sleep:151541 [120] S ==> swapper/0:0 [120]
         swapper       0 [000] 193462.410688:       sched:sched_waking: comm=sleep pid=151541 prio=120 target_cpu=000
         swapper       0 [000] 193462.410698:       sched:sched_switch: swapper/0:0 [120] R ==> sleep:151541 [120]
         :151541  151541 [000] 193462.411100: sched:sched_stat_runtime: comm=sleep pid=151541 runtime=407210 [ns]
         :151541  151541 [000] 193462.411107:       sched:sched_switch: sleep:151541 [120] X ==> swapper/0:0 [120]

But for the 'perf sched timehist --state' the process is shown in the zombie state when it is being switched out:

    # perf sched timehist --state | grep 151541
    Samples do not have callchains.
      193461.410624 [0000]  sleep[151541]                       0.000      0.054      3.129      S 
      193462.411107 [0000]  :151541[151541]                     0.000      0.000      0.408      Z 

Shouldn't both commands report the same result, or is there a reason why 'perf script' uses the dead state while 'perf sched' uses dead? It also seems that the 'perf sched' cannot read the name of the zombie process, but that can be a different issue.

Thanks,

Veronika

#1 lore raw
Quoted 23 lines · On Thu, Oct 24, 2024 at 06:36:30PM +0200, [email protected] wrote:

From: Veronika Molnarova <[email protected]>

Hello!

We encountered an issue with 'perf sched timehist --state' when we expected for a test case that the process of a workload would eventually end in the dead state(marked by 'X') and be switched out. This can be seen also by using 'perf script':

    # perf sched record -a sleep 1
    [ perf record: Woken up 1 times to write data ]
    [ perf record: Captured and wrote 0.167 MB perf.data (166 samples) ]
    # perf script | grep sleep
           sleep  151541 [000] 193461.410617: sched:sched_stat_runtime: comm=sleep pid=151541 runtime=3132275 [ns]
           sleep  151541 [000] 193461.410624:       sched:sched_switch: sleep:151541 [120] S ==> swapper/0:0 [120]
         swapper       0 [000] 193462.410688:       sched:sched_waking: comm=sleep pid=151541 prio=120 target_cpu=000
         swapper       0 [000] 193462.410698:       sched:sched_switch: swapper/0:0 [120] R ==> sleep:151541 [120]
         :151541  151541 [000] 193462.411100: sched:sched_stat_runtime: comm=sleep pid=151541 runtime=407210 [ns]
         :151541  151541 [000] 193462.411107:       sched:sched_switch: sleep:151541 [120] X ==> swapper/0:0 [120]

But for the 'perf sched timehist --state' the process is shown in the zombie state when it is being switched out:

Quoted 7 lines
    # perf sched timehist --state | grep 151541
    Samples do not have callchains.
      193461.410624 [0000]  sleep[151541]                       0.000      0.054      3.129      S 
      193462.411107 [0000]  :151541[151541]                     0.000      0.000      0.408      Z 

Shouldn't both commands report the same result, or is there a reason why 'perf script' uses the dead state while 'perf sched' uses dead?

I remember there was a discussion related to this.

https://lore.kernel.org/all/[email protected]/

And it seems it's fixed both in perf and libtraceevent. Maybe your libtraceevent is somewhat outdated and doesn't have the update.

Quoted 2 lines

It also seems that the 'perf sched' cannot read the name of the zombie process, but that can be a different issue.

Hmm.. right. It should update thread->comm using the sched_switch event.

Thanks,
Namhyung

#2 lore raw
Quoted 24 lines · Namhyung Kim
Quoted 23 lines · On Thu, Oct 24, 2024 at 06:36:30PM +0200, [email protected] wrote:

From: Veronika Molnarova <[email protected]>

Hello!

We encountered an issue with 'perf sched timehist --state' when we expected for a test case that the process of a workload would eventually end in the dead state(marked by 'X') and be switched out. This can be seen also by using 'perf script':

    # perf sched record -a sleep 1
    [ perf record: Woken up 1 times to write data ]
    [ perf record: Captured and wrote 0.167 MB perf.data (166 samples) ]
    # perf script | grep sleep
           sleep  151541 [000] 193461.410617: sched:sched_stat_runtime: comm=sleep pid=151541 runtime=3132275 [ns]
           sleep  151541 [000] 193461.410624:       sched:sched_switch: sleep:151541 [120] S ==> swapper/0:0 [120]
         swapper       0 [000] 193462.410688:       sched:sched_waking: comm=sleep pid=151541 prio=120 target_cpu=000
         swapper       0 [000] 193462.410698:       sched:sched_switch: swapper/0:0 [120] R ==> sleep:151541 [120]
         :151541  151541 [000] 193462.411100: sched:sched_stat_runtime: comm=sleep pid=151541 runtime=407210 [ns]
         :151541  151541 [000] 193462.411107:       sched:sched_switch: sleep:151541 [120] X ==> swapper/0:0 [120]

But for the 'perf sched timehist --state' the process is shown in the zombie state when it is being switched out:

Quoted 7 lines
Quoted 7 lines
    # perf sched timehist --state | grep 151541
    Samples do not have callchains.
      193461.410624 [0000]  sleep[151541]                       0.000      0.054      3.129      S 
      193462.411107 [0000]  :151541[151541]                     0.000      0.000      0.408      Z 

Shouldn't both commands report the same result, or is there a reason why 'perf script' uses the dead state while 'perf sched' uses dead?

Hello! Thanks for reacting!

Quoted 8 lines

I remember there was a discussion related to this.

https://lore.kernel.org/all/[email protected]/

And it seems it's fixed both in perf and libtraceevent. Maybe your libtraceevent is somewhat outdated and doesn't have the update.

Yes! When we've updated libtraceevent to 1.8.3, both show Z now, which is probably correct.

Then it comes into my mind whether shouldn't perf-sched and perf-script both take it from libtraceevent to unify the approaches, rather than perf-sched using its own implementation and perf-script taking it from libtraceevent... ?

Quoted 4 lines
Quoted 2 lines

It also seems that the 'perf sched' cannot read the name of the zombie process, but that can be a different issue.

Hmm.. right. It should update thread->comm using the sched_switch event.

According to the tests, the name seems that it used to be translated correctly in past, so it'd be nice if it worked again, althoug it is pretty clear which process it was, according to the PID. Thus, the test can be adjusted to search for the combination "PID, status Z", instead of "command, status Z"... I.e. to accept the current situation...

Thanks,

Michael

Quoted 5 lines

Thanks,
Namhyung

#3 lore raw
Quoted 37 lines · Michael Petlan
Quoted 24 lines · Namhyung Kim
Quoted 23 lines · On Thu, Oct 24, 2024 at 06:36:30PM +0200, [email protected] wrote:

From: Veronika Molnarova <[email protected]>

Hello!

We encountered an issue with 'perf sched timehist --state' when we expected for a test case that the process of a workload would eventually end in the dead state(marked by 'X') and be switched out. This can be seen also by using 'perf script':

    # perf sched record -a sleep 1
    [ perf record: Woken up 1 times to write data ]
    [ perf record: Captured and wrote 0.167 MB perf.data (166 samples) ]
    # perf script | grep sleep
           sleep  151541 [000] 193461.410617: sched:sched_stat_runtime: comm=sleep pid=151541 runtime=3132275 [ns]
           sleep  151541 [000] 193461.410624:       sched:sched_switch: sleep:151541 [120] S ==> swapper/0:0 [120]
         swapper       0 [000] 193462.410688:       sched:sched_waking: comm=sleep pid=151541 prio=120 target_cpu=000
         swapper       0 [000] 193462.410698:       sched:sched_switch: swapper/0:0 [120] R ==> sleep:151541 [120]
         :151541  151541 [000] 193462.411100: sched:sched_stat_runtime: comm=sleep pid=151541 runtime=407210 [ns]
         :151541  151541 [000] 193462.411107:       sched:sched_switch: sleep:151541 [120] X ==> swapper/0:0 [120]

But for the 'perf sched timehist --state' the process is shown in the zombie state when it is being switched out:

Quoted 7 lines
Quoted 7 lines
    # perf sched timehist --state | grep 151541
    Samples do not have callchains.
      193461.410624 [0000]  sleep[151541]                       0.000      0.054      3.129      S 
      193462.411107 [0000]  :151541[151541]                     0.000      0.000      0.408      Z 

Shouldn't both commands report the same result, or is there a reason why 'perf script' uses the dead state while 'perf sched' uses dead?

Hello! Thanks for reacting!

Quoted 3 lines

I remember there was a discussion related to this.

Quoted 7 lines
Quoted 5 lines

https://lore.kernel.org/all/[email protected]/

And it seems it's fixed both in perf and libtraceevent. Maybe your libtraceevent is somewhat outdated and doesn't have the update.

Yes! When we've updated libtraceevent to 1.8.3, both show Z now, which is probably correct.

Cool.

Quoted 5 lines

Then it comes into my mind whether shouldn't perf-sched and perf-script both take it from libtraceevent to unify the approaches, rather than perf-sched using its own implementation and perf-script taking it from libtraceevent... ?

Yeah, I think it's better to use libtraceevent for it.

Quoted 11 lines
Quoted 5 lines
Quoted 3 lines

It also seems that the 'perf sched' cannot read the name of the zombie process, but that can be a different issue.

Hmm.. right. It should update thread->comm using the sched_switch event.

According to the tests, the name seems that it used to be translated correctly in past, so it'd be nice if it worked again, althoug it is pretty clear which process it was, according to the PID. Thus, the test can be adjusted to search for the combination "PID, status Z", instead of "command, status Z"... I.e. to accept the current situation...

Sounds good. Please send the patch.

Thanks,
Namhyung