TREE_SRCU slows hotplug by factor ~16
Greetings,
Running Steven's hotplug stress script in tip w. CLASSIC_SRCU takes 55s in my i4790 box, whereas TREE_SRCU takes over 16m. (Master with the same config does it in 39s.. but then lockdep isn't enabled in master)
-Mike
Quoted 5 lines · Mike Galbraith
Greetings,
Running Steven's hotplug stress script in tip w.
CLASSIC_SRCUtakes 55s in my i4790 box, whereasTREE_SRCUtakes over 16m. (Master with the same config does it in 39s.. but then lockdep isn't enabled in master)
That doesn't sound good...
But exactly which version are you using? There were some 8x slowdowns earlier this week that have since been fixed. Also, how many CPUs are on your system?
Thanx, Paul
Quoted 7 lines · Paul E. McKenney
Quoted 5 lines · Mike Galbraith
Greetings,
Running Steven's hotplug stress script in tip w.
CLASSIC_SRCUtakes 55s in my i4790 box, whereasTREE_SRCUtakes over 16m. (Master with the same config does it in 39s.. but then lockdep isn't enabled in master)
Quoted 5 lines
That doesn't sound good...
But exactly which version are you using? There were some 8x slowdowns earlier this week that have since been fixed. Also, how many CPUs are on your system?
4 cores+SMT, tip.yesterday_afternoon (precise!) was initial test point, tip.this_morning (v4.11-rc7-955-g76f8909e1cd8) is still slow.
-Mike
Quoted 8 lines · Mike Galbraith
Quoted 7 lines · Paul E. McKenney
Quoted 5 lines · Mike Galbraith
Greetings,
Running Steven's hotplug stress script in tip w.
CLASSIC_SRCUtakes 55s in my i4790 box, whereasTREE_SRCUtakes over 16m. (Master with the same config does it in 39s.. but then lockdep isn't enabled in master)
Quoted 8 lines
Quoted 5 lines
That doesn't sound good...
But exactly which version are you using? There were some 8x slowdowns earlier this week that have since been fixed. Also, how many CPUs are on your system?
4 cores+SMT, tip.yesterday_afternoon (precise!) was initial test point, tip.this_morning (v4.11-rc7-955-g76f8909e1cd8) is still slow.
OK, that is after the earlier fix! And modest in size as well.
Assuming 4ms per TREE_SRCU synchronize_srcu(), I end up with the CLASSIC_SRCU doing a synchronize_srcu() every 200us or so, which I am having a hard time believing. But it does matter, because it controls what approach I take to fixing this.
Could you please collect an ftrace (or whatever) showing the timestamp sequence of calls to synchronize_srcu(), synchronize_srcu_expedited(), and call_srcu() during the execution of the stress script? If it is easy to do, also the timestamp sequence of returns from synchronize_srcu() and synchronize_srcu_expedited()?
Thanx, Paul
Quoted 5 lines · Paul E. McKenney
Could you please collect an ftrace (or whatever) showing the timestamp sequence of calls to
synchronize_srcu(),synchronize_srcu_expedited(), andcall_srcu()during the execution of the stress script? If it is easy to do, also the timestamp sequence of returns fromsynchronize_srcu()andsynchronize_srcu_expedited()?
The first two minutes below config bits, trace_printk([enter/exit]) in each function. If you (unlikely, but..) want the whole trace, holler.
# RCU SubsystemCONFIG_TREE_RCU=yCONFIG_RCU_EXPERT=yCONFIG_SRCU=y
# CONFIG_CLASSIC_SRCU is not setCONFIG_TREE_SRCU=y
# CONFIG_TASKS_RCU is not setCONFIG_RCU_STALL_COMMON=yCONFIG_RCU_FANOUT=64CONFIG_RCU_FANOUT_LEAF=16
# CONFIG_RCU_FAST_NO_HZ is not set
# CONFIG_TREE_RCU_TRACE is not setCONFIG_RCU_KTHREAD_PRIO=0CONFIG_RCU_NOCB_CPU=yCONFIG_RCU_NOCB_CPU_NONE=y
# CONFIG_RCU_NOCB_CPU_ZERO is not set
# CONFIG_RCU_NOCB_CPU_ALL is not set
# RCU Debugging
# CONFIG_PROVE_RCU is not set
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_RCU_PERF_TEST is not set
# CONFIG_RCU_TORTURE_TEST is not setCONFIG_RCU_CPU_STALL_TIMEOUT=60
# CONFIG_RCU_TRACE is not set
# CONFIG_RCU_EQS_DEBUG is not set
# tracer: nop
#
# entries-in-buffer/entries-written: 229332/229332 #P:8
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | stress-cpu-hotp-2443 [004] .... 144.123608: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.123610: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.123612: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.139528: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.139646: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.139646: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.139648: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.155813: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.155981: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.155982: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.155983: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.172078: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.172234: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.172235: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.172237: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.188319: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.188465: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.188466: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.188468: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.202528: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.202650: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.202651: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.202652: call_srcu: exit stress-cpu-hotp-2443 [003] .... 144.219081: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 144.219238: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 144.219238: call_srcu: enter stress-cpu-hotp-2443 [003] .... 144.219240: call_srcu: exit stress-cpu-hotp-2443 [004] .... 144.235319: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 144.235567: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.235568: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.235571: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.251548: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.251706: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.251706: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.251708: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.267865: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.268132: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.268133: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.268136: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.284078: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.284257: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.284258: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.284260: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.300314: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.300462: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.300463: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.300464: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.314533: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.314657: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.314658: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.314659: call_srcu: exit stress-cpu-hotp-2443 [003] .... 144.331056: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 144.331227: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 144.331227: call_srcu: enter stress-cpu-hotp-2443 [003] .... 144.331229: call_srcu: exit stress-cpu-hotp-2443 [004] .... 144.347334: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 144.347517: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.347518: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.347519: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.363567: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.363731: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.363732: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.363733: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.379804: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.379987: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.379988: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.379989: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.396077: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.396259: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.396260: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.396261: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.412385: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.412653: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.412654: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.412657: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.426647: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.426888: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.426889: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.426891: call_srcu: exit stress-cpu-hotp-2443 [003] .... 144.443134: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 144.443325: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 144.443325: call_srcu: enter stress-cpu-hotp-2443 [003] .... 144.443327: call_srcu: exit stress-cpu-hotp-2443 [004] .... 144.459327: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 144.459521: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.459522: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.459524: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.475553: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.475672: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.475673: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.475674: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.491798: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.491919: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.491920: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.491921: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.508035: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.508160: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.508161: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.508162: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.524301: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.524419: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.524419: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.524421: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.538545: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.538665: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.538666: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.538667: call_srcu: exit stress-cpu-hotp-2443 [003] .... 144.555051: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 144.555174: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 144.555174: call_srcu: enter stress-cpu-hotp-2443 [003] .... 144.555175: call_srcu: exit stress-cpu-hotp-2443 [004] .... 144.571290: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 144.571412: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.571412: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.571413: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.587554: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.587672: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.587673: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.587674: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.603843: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.604008: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.604008: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.604010: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.620087: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.620317: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.620318: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.620321: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.636342: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.636520: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.636521: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.636523: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.650579: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.650737: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.650738: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.650739: call_srcu: exit stress-cpu-hotp-2443 [003] .... 144.667084: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 144.667247: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 144.667247: call_srcu: enter stress-cpu-hotp-2443 [003] .... 144.667249: call_srcu: exit stress-cpu-hotp-2443 [004] .... 144.683340: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 144.683513: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.683514: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.683516: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.699556: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.699675: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.699675: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.699676: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.715841: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.716013: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.716014: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.716015: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.732096: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.732269: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.732270: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.732271: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.748393: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.748641: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.748642: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.748645: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.762639: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.762854: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.762855: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.762857: call_srcu: exit stress-cpu-hotp-2443 [003] .... 144.779100: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 144.779278: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 144.779279: call_srcu: enter stress-cpu-hotp-2443 [003] .... 144.779280: call_srcu: exit stress-cpu-hotp-2443 [004] .... 144.795348: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 144.795525: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.795526: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.795527: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.811583: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.811749: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.811750: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.811751: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.827831: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.827997: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.827998: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.827999: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.844104: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.844281: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.844281: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.844283: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.860339: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.860484: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.860485: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.860487: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.874584: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.874737: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.874737: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.874739: call_srcu: exit stress-cpu-hotp-2443 [003] .... 144.891117: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 144.891459: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 144.891460: call_srcu: enter stress-cpu-hotp-2443 [003] .... 144.891463: call_srcu: exit stress-cpu-hotp-2443 [004] .... 144.907376: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 144.907628: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.907629: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.907631: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.923590: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.923756: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.923756: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.923758: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.939837: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.940007: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.940008: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.940009: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.956110: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.956285: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.956286: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.956287: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.972344: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.972492: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.972493: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.972495: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.986566: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.986689: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.986689: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.986690: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.003114: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.003292: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.003293: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.003294: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.019350: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.019528: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.019529: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.019531: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.035659: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.035939: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.035940: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.035943: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.051891: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.052106: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.052107: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.052109: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.068125: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.068279: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.068279: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.068280: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.084332: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.084472: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.084473: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.084474: call_srcu: exit stress-cpu-hotp-2443 [002] .... 145.098605: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 145.098743: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 145.098744: call_srcu: enter stress-cpu-hotp-2443 [002] .... 145.098745: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.114590: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.114728: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.114728: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.114729: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.130591: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.130728: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.130728: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.130730: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.147857: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.228078: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.228079: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.228080: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.244025: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.244106: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.244106: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.244107: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.260286: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.260361: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.260361: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.260362: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.274522: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.274594: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.274594: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.274595: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.291289: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.291358: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.291358: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.291359: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.307529: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.307599: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.307599: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.307600: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.323901: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.324063: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.324063: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.324065: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.340122: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.340288: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.340289: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.340291: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.356378: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.356519: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.356520: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.356521: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.370650: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.370797: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.370798: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.370799: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.387385: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.387547: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.387548: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.387549: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.403636: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.403799: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.403800: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.403802: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.419885: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.420055: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.420056: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.420057: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.436154: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.436372: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.436372: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.436374: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.452413: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.452550: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.452551: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.452552: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.466606: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.466721: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.466721: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.466722: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.483395: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.483555: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.483555: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.483557: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.499628: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.499761: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .N.. 145.499774: call_srcu: enter stress-cpu-hotp-2443 [005] .N.. 145.499775: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.515879: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.516009: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.516009: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.516011: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.532164: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.532488: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.532489: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.532492: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.548447: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.548659: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.548660: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.548663: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.562706: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.562897: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.562898: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.562900: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.579403: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.579562: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.579563: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.579564: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.595636: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.595766: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.595766: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.595768: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.611902: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.612059: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.612060: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.612061: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.628136: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.628266: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.628267: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.628268: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.644392: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.644518: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.644519: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.644521: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.658609: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.658698: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.658699: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.658700: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.675410: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.675570: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.675571: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.675572: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.691706: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.691950: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.691951: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.691953: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.707926: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.708096: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.708097: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.708099: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.724154: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.724333: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.724333: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.724335: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.740412: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.740559: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.740559: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.740561: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.754685: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.754835: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.754836: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.754838: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.771415: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.771591: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.771592: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.771594: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.787635: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.787747: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.787748: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.787749: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.803943: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.804168: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.804170: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.804172: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.820168: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.820320: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.820321: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.820322: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.836412: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.836543: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.836544: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.836545: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.850638: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.850757: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.850757: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.850758: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.867433: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.867597: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.867598: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.867599: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.883680: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.883843: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.883843: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.883845: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.899908: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.900073: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.900073: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.900075: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.916213: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.916405: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.916406: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.916408: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.932453: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.932653: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.932653: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.932656: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.946674: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.946818: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.946819: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.946820: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.963402: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.963539: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.963540: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.963542: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.979671: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.979790: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.979790: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.979792: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.995927: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.996046: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.996046: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.996048: call_srcu: exit stress-cpu-hotp-2443 [007] .... 146.012167: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 146.012282: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 146.012283: call_srcu: enter stress-cpu-hotp-2443 [007] .... 146.012284: call_srcu: exit stress-cpu-hotp-2443 [000] .... 146.028405: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 146.028522: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 146.028522: call_srcu: enter stress-cpu-hotp-2443 [000] .... 146.028523: call_srcu: exit stress-cpu-hotp-2443 [003] .... 146.042653: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 146.042774: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 146.042774: call_srcu: enter stress-cpu-hotp-2443 [003] .... 146.042775: call_srcu: exit stress-cpu-hotp-2443 [004] .... 146.059407: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 146.059521: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 146.059522: call_srcu: enter stress-cpu-hotp-2443 [004] .... 146.059523: call_srcu: exit stress-cpu-hotp-2443 [005] .... 146.075658: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 146.075772: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 146.075773: call_srcu: enter stress-cpu-hotp-2443 [005] .... 146.075774: call_srcu: exit stress-cpu-hotp-2443 [006] .... 146.091906: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 146.092021: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 146.092022: call_srcu: enter stress-cpu-hotp-2443 [006] .... 146.092023: call_srcu: exit stress-cpu-hotp-2443 [007] .... 146.108187: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 146.108336: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 146.108337: call_srcu: enter stress-cpu-hotp-2443 [007] .... 146.108338: call_srcu: exit stress-cpu-hotp-2443 [000] .... 146.124439: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 146.124572: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 146.124572: call_srcu: enter stress-cpu-hotp-2443 [000] .... 146.124574: call_srcu: exit stress-cpu-hotp-2443 [003] .... 146.138656: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 146.138773: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 146.138773: call_srcu: enter stress-cpu-hotp-2443 [003] .... 146.138774: call_srcu: exit
Quoted 6 lines · Mike Galbraith
# tracer: nop
#
# entries-in-buffer/entries-written: 229332/229332 #P:8
#
# _-----=> irqs-off
# / _----=> need-resched
Quoted 9 lines
# | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | stress-cpu-hotp-2443 [004] .... 144.123608: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.123610: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.123612: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.139528: synchronize_srcu: exit
I probably could have sent a LOT less trace.
classic:
# tracer: nop
#
# entries-in-buffer/entries-written: 114688/114688 #P:8
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | stress-cpu-hotp-4337 [007] .... 174.082801: synchronize_srcu: enter stress-cpu-hotp-4337 [007] .... 174.082802: synchronize_srcu: exit stress-cpu-hotp-4337 [007] .... 174.082896: synchronize_srcu: enter stress-cpu-hotp-4337 [007] .... 174.082896: synchronize_srcu: exit stress-cpu-hotp-4337 [007] .... 174.082975: synchronize_srcu: enter stress-cpu-hotp-4337 [007] .... 174.082975: synchronize_srcu: exit stress-cpu-hotp-4337 [007] .... 174.083051: synchronize_srcu: enter stress-cpu-hotp-4337 [007] .... 174.083051: synchronize_srcu: exit
....
Quoted 28 lines · Mike Galbraith
Quoted 5 lines · Paul E. McKenney
Could you please collect an ftrace (or whatever) showing the timestamp sequence of calls to
synchronize_srcu(),synchronize_srcu_expedited(), andcall_srcu()during the execution of the stress script? If it is easy to do, also the timestamp sequence of returns fromsynchronize_srcu()andsynchronize_srcu_expedited()?The first two minutes below config bits,
trace_printk([enter/exit])in each function. If you (unlikely, but..) want the whole trace, holler.# RCU Subsystem
CONFIG_TREE_RCU=yCONFIG_RCU_EXPERT=yCONFIG_SRCU=y
#CONFIG_CLASSIC_SRCUis not setCONFIG_TREE_SRCU=y
#CONFIG_TASKS_RCUis not setCONFIG_RCU_STALL_COMMON=yCONFIG_RCU_FANOUT=64CONFIG_RCU_FANOUT_LEAF=16
#CONFIG_RCU_FAST_NO_HZis not set
#CONFIG_TREE_RCU_TRACEis not setCONFIG_RCU_KTHREAD_PRIO=0CONFIG_RCU_NOCB_CPU=yCONFIG_RCU_NOCB_CPU_NONE=y
#CONFIG_RCU_NOCB_CPU_ZEROis not set
#CONFIG_RCU_NOCB_CPU_ALLis not set
Quoted 8 lines
# RCU Debugging
#CONFIG_PROVE_RCUis not set
#CONFIG_SPARSE_RCU_POINTERis not set
#CONFIG_RCU_PERF_TESTis not set
#CONFIG_RCU_TORTURE_TESTis not setCONFIG_RCU_CPU_STALL_TIMEOUT=60
#CONFIG_RCU_TRACEis not set
#CONFIG_RCU_EQS_DEBUGis not set
Thank you for the data, extremely helpful! I now know that I must immediately fix this the hard way rather than trying several simpler possibilities that, according to your trace data, would be complete wastes of time.
My initial calculation was actually optimistic. There was one exit-to-enter interval at about 80 milliseconds, but the second longest is 342 microseconds, and the vast majority are under 200 microseconds.
Back to the drawing board!
This will take some time, but I should have a patch for you in a few days, hopefully sooner.
Thanx, Paul
Quoted 500 lines
# tracer: nop
#
# entries-in-buffer/entries-written: 229332/229332 #P:8
#
# _-----=> irqs-off
# / _----=> need-resched# | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | stress-cpu-hotp-2443 [004] .... 144.123608: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.123610: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.123612: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.139528: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.139646: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.139646: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.139648: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.155813: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.155981: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.155982: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.155983: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.172078: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.172234: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.172235: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.172237: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.188319: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.188465: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.188466: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.188468: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.202528: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.202650: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.202651: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.202652: call_srcu: exit stress-cpu-hotp-2443 [003] .... 144.219081: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 144.219238: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 144.219238: call_srcu: enter stress-cpu-hotp-2443 [003] .... 144.219240: call_srcu: exit stress-cpu-hotp-2443 [004] .... 144.235319: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 144.235567: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.235568: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.235571: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.251548: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.251706: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.251706: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.251708: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.267865: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.268132: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.268133: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.268136: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.284078: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.284257: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.284258: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.284260: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.300314: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.300462: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.300463: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.300464: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.314533: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.314657: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.314658: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.314659: call_srcu: exit stress-cpu-hotp-2443 [003] .... 144.331056: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 144.331227: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 144.331227: call_srcu: enter stress-cpu-hotp-2443 [003] .... 144.331229: call_srcu: exit stress-cpu-hotp-2443 [004] .... 144.347334: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 144.347517: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.347518: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.347519: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.363567: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.363731: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.363732: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.363733: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.379804: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.379987: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.379988: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.379989: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.396077: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.396259: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.396260: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.396261: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.412385: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.412653: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.412654: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.412657: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.426647: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.426888: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.426889: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.426891: call_srcu: exit stress-cpu-hotp-2443 [003] .... 144.443134: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 144.443325: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 144.443325: call_srcu: enter stress-cpu-hotp-2443 [003] .... 144.443327: call_srcu: exit stress-cpu-hotp-2443 [004] .... 144.459327: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 144.459521: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.459522: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.459524: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.475553: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.475672: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.475673: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.475674: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.491798: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.491919: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.491920: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.491921: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.508035: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.508160: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.508161: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.508162: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.524301: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.524419: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.524419: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.524421: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.538545: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.538665: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.538666: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.538667: call_srcu: exit stress-cpu-hotp-2443 [003] .... 144.555051: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 144.555174: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 144.555174: call_srcu: enter stress-cpu-hotp-2443 [003] .... 144.555175: call_srcu: exit stress-cpu-hotp-2443 [004] .... 144.571290: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 144.571412: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.571412: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.571413: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.587554: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.587672: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.587673: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.587674: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.603843: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.604008: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.604008: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.604010: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.620087: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.620317: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.620318: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.620321: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.636342: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.636520: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.636521: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.636523: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.650579: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.650737: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.650738: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.650739: call_srcu: exit stress-cpu-hotp-2443 [003] .... 144.667084: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 144.667247: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 144.667247: call_srcu: enter stress-cpu-hotp-2443 [003] .... 144.667249: call_srcu: exit stress-cpu-hotp-2443 [004] .... 144.683340: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 144.683513: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.683514: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.683516: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.699556: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.699675: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.699675: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.699676: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.715841: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.716013: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.716014: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.716015: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.732096: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.732269: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.732270: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.732271: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.748393: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.748641: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.748642: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.748645: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.762639: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.762854: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.762855: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.762857: call_srcu: exit stress-cpu-hotp-2443 [003] .... 144.779100: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 144.779278: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 144.779279: call_srcu: enter stress-cpu-hotp-2443 [003] .... 144.779280: call_srcu: exit stress-cpu-hotp-2443 [004] .... 144.795348: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 144.795525: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.795526: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.795527: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.811583: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.811749: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.811750: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.811751: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.827831: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.827997: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.827998: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.827999: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.844104: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.844281: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.844281: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.844283: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.860339: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.860484: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.860485: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.860487: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.874584: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.874737: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.874737: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.874739: call_srcu: exit stress-cpu-hotp-2443 [003] .... 144.891117: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 144.891459: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 144.891460: call_srcu: enter stress-cpu-hotp-2443 [003] .... 144.891463: call_srcu: exit stress-cpu-hotp-2443 [004] .... 144.907376: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 144.907628: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 144.907629: call_srcu: enter stress-cpu-hotp-2443 [004] .... 144.907631: call_srcu: exit stress-cpu-hotp-2443 [005] .... 144.923590: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 144.923756: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 144.923756: call_srcu: enter stress-cpu-hotp-2443 [005] .... 144.923758: call_srcu: exit stress-cpu-hotp-2443 [006] .... 144.939837: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 144.940007: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 144.940008: call_srcu: enter stress-cpu-hotp-2443 [006] .... 144.940009: call_srcu: exit stress-cpu-hotp-2443 [007] .... 144.956110: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 144.956285: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 144.956286: call_srcu: enter stress-cpu-hotp-2443 [007] .... 144.956287: call_srcu: exit stress-cpu-hotp-2443 [000] .... 144.972344: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 144.972492: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 144.972493: call_srcu: enter stress-cpu-hotp-2443 [000] .... 144.972495: call_srcu: exit stress-cpu-hotp-2443 [002] .... 144.986566: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 144.986689: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 144.986689: call_srcu: enter stress-cpu-hotp-2443 [002] .... 144.986690: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.003114: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.003292: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.003293: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.003294: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.019350: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.019528: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.019529: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.019531: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.035659: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.035939: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.035940: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.035943: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.051891: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.052106: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.052107: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.052109: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.068125: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.068279: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.068279: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.068280: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.084332: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.084472: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.084473: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.084474: call_srcu: exit stress-cpu-hotp-2443 [002] .... 145.098605: synchronize_srcu: exit stress-cpu-hotp-2443 [002] .... 145.098743: synchronize_srcu: enter stress-cpu-hotp-2443 [002] .... 145.098744: call_srcu: enter stress-cpu-hotp-2443 [002] .... 145.098745: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.114590: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.114728: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.114728: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.114729: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.130591: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.130728: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.130728: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.130730: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.147857: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.228078: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.228079: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.228080: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.244025: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.244106: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.244106: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.244107: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.260286: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.260361: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.260361: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.260362: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.274522: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.274594: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.274594: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.274595: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.291289: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.291358: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.291358: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.291359: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.307529: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.307599: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.307599: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.307600: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.323901: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.324063: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.324063: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.324065: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.340122: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.340288: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.340289: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.340291: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.356378: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.356519: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.356520: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.356521: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.370650: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.370797: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.370798: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.370799: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.387385: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.387547: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.387548: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.387549: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.403636: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.403799: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.403800: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.403802: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.419885: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.420055: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.420056: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.420057: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.436154: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.436372: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.436372: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.436374: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.452413: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.452550: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.452551: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.452552: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.466606: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.466721: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.466721: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.466722: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.483395: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.483555: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.483555: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.483557: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.499628: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.499761: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .N.. 145.499774: call_srcu: enter stress-cpu-hotp-2443 [005] .N.. 145.499775: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.515879: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.516009: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.516009: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.516011: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.532164: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.532488: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.532489: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.532492: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.548447: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.548659: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.548660: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.548663: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.562706: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.562897: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.562898: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.562900: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.579403: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.579562: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.579563: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.579564: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.595636: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.595766: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.595766: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.595768: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.611902: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.612059: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.612060: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.612061: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.628136: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.628266: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.628267: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.628268: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.644392: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.644518: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.644519: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.644521: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.658609: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.658698: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.658699: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.658700: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.675410: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.675570: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.675571: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.675572: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.691706: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.691950: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.691951: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.691953: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.707926: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.708096: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.708097: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.708099: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.724154: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.724333: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.724333: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.724335: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.740412: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.740559: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.740559: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.740561: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.754685: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.754835: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.754836: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.754838: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.771415: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.771591: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.771592: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.771594: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.787635: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.787747: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.787748: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.787749: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.803943: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.804168: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.804170: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.804172: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.820168: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.820320: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.820321: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.820322: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.836412: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.836543: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.836544: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.836545: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.850638: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.850757: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.850757: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.850758: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.867433: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.867597: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.867598: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.867599: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.883680: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.883843: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.883843: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.883845: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.899908: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.900073: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.900073: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.900075: call_srcu: exit stress-cpu-hotp-2443 [007] .... 145.916213: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 145.916405: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 145.916406: call_srcu: enter stress-cpu-hotp-2443 [007] .... 145.916408: call_srcu: exit stress-cpu-hotp-2443 [000] .... 145.932453: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 145.932653: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 145.932653: call_srcu: enter stress-cpu-hotp-2443 [000] .... 145.932656: call_srcu: exit stress-cpu-hotp-2443 [003] .... 145.946674: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 145.946818: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 145.946819: call_srcu: enter stress-cpu-hotp-2443 [003] .... 145.946820: call_srcu: exit stress-cpu-hotp-2443 [004] .... 145.963402: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 145.963539: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 145.963540: call_srcu: enter stress-cpu-hotp-2443 [004] .... 145.963542: call_srcu: exit stress-cpu-hotp-2443 [005] .... 145.979671: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 145.979790: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 145.979790: call_srcu: enter stress-cpu-hotp-2443 [005] .... 145.979792: call_srcu: exit stress-cpu-hotp-2443 [006] .... 145.995927: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 145.996046: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 145.996046: call_srcu: enter stress-cpu-hotp-2443 [006] .... 145.996048: call_srcu: exit stress-cpu-hotp-2443 [007] .... 146.012167: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 146.012282: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 146.012283: call_srcu: enter stress-cpu-hotp-2443 [007] .... 146.012284: call_srcu: exit stress-cpu-hotp-2443 [000] .... 146.028405: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 146.028522: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 146.028522: call_srcu: enter stress-cpu-hotp-2443 [000] .... 146.028523: call_srcu: exit stress-cpu-hotp-2443 [003] .... 146.042653: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 146.042774: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 146.042774: call_srcu: enter stress-cpu-hotp-2443 [003] .... 146.042775: call_srcu: exit stress-cpu-hotp-2443 [004] .... 146.059407: synchronize_srcu: exit stress-cpu-hotp-2443 [004] .... 146.059521: synchronize_srcu: enter stress-cpu-hotp-2443 [004] .... 146.059522: call_srcu: enter stress-cpu-hotp-2443 [004] .... 146.059523: call_srcu: exit stress-cpu-hotp-2443 [005] .... 146.075658: synchronize_srcu: exit stress-cpu-hotp-2443 [005] .... 146.075772: synchronize_srcu: enter stress-cpu-hotp-2443 [005] .... 146.075773: call_srcu: enter stress-cpu-hotp-2443 [005] .... 146.075774: call_srcu: exit stress-cpu-hotp-2443 [006] .... 146.091906: synchronize_srcu: exit stress-cpu-hotp-2443 [006] .... 146.092021: synchronize_srcu: enter stress-cpu-hotp-2443 [006] .... 146.092022: call_srcu: enter stress-cpu-hotp-2443 [006] .... 146.092023: call_srcu: exit stress-cpu-hotp-2443 [007] .... 146.108187: synchronize_srcu: exit stress-cpu-hotp-2443 [007] .... 146.108336: synchronize_srcu: enter stress-cpu-hotp-2443 [007] .... 146.108337: call_srcu: enter stress-cpu-hotp-2443 [007] .... 146.108338: call_srcu: exit stress-cpu-hotp-2443 [000] .... 146.124439: synchronize_srcu: exit stress-cpu-hotp-2443 [000] .... 146.124572: synchronize_srcu: enter stress-cpu-hotp-2443 [000] .... 146.124572: call_srcu: enter stress-cpu-hotp-2443 [000] .... 146.124574: call_srcu: exit stress-cpu-hotp-2443 [003] .... 146.138656: synchronize_srcu: exit stress-cpu-hotp-2443 [003] .... 146.138773: synchronize_srcu: enter stress-cpu-hotp-2443 [003] .... 146.138773: call_srcu: enter stress-cpu-hotp-2443 [003] .... 146.138774: call_srcu: exit
Quoted 48 lines · Paul E. McKenney
Quoted 28 lines · Mike Galbraith
Quoted 5 lines · Paul E. McKenney
Could you please collect an ftrace (or whatever) showing the timestamp sequence of calls to
synchronize_srcu(),synchronize_srcu_expedited(), andcall_srcu()during the execution of the stress script? If it is easy to do, also the timestamp sequence of returns fromsynchronize_srcu()andsynchronize_srcu_expedited()?The first two minutes below config bits,
trace_printk([enter/exit])in each function. If you (unlikely, but..) want the whole trace, holler.# RCU Subsystem
CONFIG_TREE_RCU=yCONFIG_RCU_EXPERT=yCONFIG_SRCU=y
#CONFIG_CLASSIC_SRCUis not setCONFIG_TREE_SRCU=y
#CONFIG_TASKS_RCUis not setCONFIG_RCU_STALL_COMMON=yCONFIG_RCU_FANOUT=64CONFIG_RCU_FANOUT_LEAF=16
#CONFIG_RCU_FAST_NO_HZis not set
#CONFIG_TREE_RCU_TRACEis not setCONFIG_RCU_KTHREAD_PRIO=0CONFIG_RCU_NOCB_CPU=yCONFIG_RCU_NOCB_CPU_NONE=y
#CONFIG_RCU_NOCB_CPU_ZEROis not set
#CONFIG_RCU_NOCB_CPU_ALLis not setQuoted 8 lines
# RCU Debugging
#CONFIG_PROVE_RCUis not set
#CONFIG_SPARSE_RCU_POINTERis not set
#CONFIG_RCU_PERF_TESTis not set
#CONFIG_RCU_TORTURE_TESTis not setCONFIG_RCU_CPU_STALL_TIMEOUT=60
#CONFIG_RCU_TRACEis not set
#CONFIG_RCU_EQS_DEBUGis not setThank you for the data, extremely helpful! I now know that I must immediately fix this the hard way rather than trying several simpler possibilities that, according to your trace data, would be complete wastes of time.
My initial calculation was actually optimistic. There was one exit-to-enter interval at about 80 milliseconds, but the second longest is 342 microseconds, and the vast majority are under 200 microseconds.
Quoted 4 lines
Back to the drawing board!
This will take some time, but I should have a patch for you in a few days, hopefully sooner.
Making progress, might have something late tomorrow (Wednesday) Pacific Time, will definitely have something Thursday Pacific Time.
Thanx, Paul
Quoted 49 lines · Paul E. McKenney
Quoted 48 lines · Paul E. McKenney
Quoted 28 lines · Mike Galbraith
Quoted 5 lines · Paul E. McKenney
Could you please collect an ftrace (or whatever) showing the timestamp sequence of calls to
synchronize_srcu(),synchronize_srcu_expedited(), andcall_srcu()during the execution of the stress script? If it is easy to do, also the timestamp sequence of returns fromsynchronize_srcu()andsynchronize_srcu_expedited()?The first two minutes below config bits,
trace_printk([enter/exit])in each function. If you (unlikely, but..) want the whole trace, holler.# RCU Subsystem
CONFIG_TREE_RCU=yCONFIG_RCU_EXPERT=yCONFIG_SRCU=y
#CONFIG_CLASSIC_SRCUis not setCONFIG_TREE_SRCU=y
#CONFIG_TASKS_RCUis not setCONFIG_RCU_STALL_COMMON=yCONFIG_RCU_FANOUT=64CONFIG_RCU_FANOUT_LEAF=16
#CONFIG_RCU_FAST_NO_HZis not set
#CONFIG_TREE_RCU_TRACEis not setCONFIG_RCU_KTHREAD_PRIO=0CONFIG_RCU_NOCB_CPU=yCONFIG_RCU_NOCB_CPU_NONE=y
#CONFIG_RCU_NOCB_CPU_ZEROis not set
#CONFIG_RCU_NOCB_CPU_ALLis not setQuoted 8 lines
# RCU Debugging
#CONFIG_PROVE_RCUis not set
#CONFIG_SPARSE_RCU_POINTERis not set
#CONFIG_RCU_PERF_TESTis not set
#CONFIG_RCU_TORTURE_TESTis not setCONFIG_RCU_CPU_STALL_TIMEOUT=60
#CONFIG_RCU_TRACEis not set
#CONFIG_RCU_EQS_DEBUGis not setThank you for the data, extremely helpful! I now know that I must immediately fix this the hard way rather than trying several simpler possibilities that, according to your trace data, would be complete wastes of time.
My initial calculation was actually optimistic. There was one exit-to-enter interval at about 80 milliseconds, but the second longest is 342 microseconds, and the vast majority are under 200 microseconds.
Quoted 7 lines
Quoted 4 lines
Back to the drawing board!
This will take some time, but I should have a patch for you in a few days, hopefully sooner.
Making progress, might have something late tomorrow (Wednesday) Pacific Time, will definitely have something Thursday Pacific Time.
And a sneak preview, semi-tested. If you get a chance to run this, please let me know now it goes.
This should apply on -tip or on -rcu. Probably on recent -next as well.
Thanx, Paul
------------------------------------------------------------------------
Documentation/admin-guide/kernel-parameters.txt +8 -0
@@ -3779,6 +3779,14 @@ spia_pedr= spia_peddr= + srcutree.exp_holdoff [KNL] + Specifies how many nanoseconds must elapse + since the end of the last SRCU grace period for + a given srcu_struct until the next normal SRCU + grace period will be considered for automatic + expediting. Set to zero to disable automatic + expediting. + stacktrace [FTRACE] Enabled the stack tracer on boot up.
include/linux/srcuclassic.h +14 -0
@@ -98,4 +98,18 @@ void synchronize_srcu_expedited(struct srcu_struct *sp);void srcu_barrier (struct srcu_struct *sp);unsigned long srcu_batches_completed (struct srcu_struct *sp); +static inline void srcutorture_get_gp_data (enum rcutorture_type test_type, +struct srcu_struct *sp,int *flags, +unsigned long *gpnum, +unsigned long *completed) +{ +if (test_type !=SRCU_FLAVOR ) +return ; + *flags =0 ; + *completed = sp->completed; + *gpnum = *completed; +if (sp->batch_queue.head || sp->batch_check0.head || sp->batch_check0.head) + (*gpnum)++; +} +#endif
include/linux/srcutiny.h +12 -0
@@ -78,4 +78,16 @@ static inline unsigned long srcu_batches_completed(struct srcu_struct *sp) return 0; } +static inline void srcutorture_get_gp_data (enum rcutorture_type test_type, +struct srcu_struct *sp,int *flags, +unsigned long *gpnum, +unsigned long *completed) +{ +if (test_type !=SRCU_FLAVOR ) +return ; + *flags =0 ; + *completed = sp->srcu_gp_seq; + *gpnum = *completed; +} +#endif
include/linux/srcutree.h +12 -1
@@ -43,10 +43,13 @@ struct srcu_data {spinlock_t lock____cacheline_internodealigned_in_smp ;struct rcu_segcblist srcu_cblist;/* List of callbacks.*/ unsigned long srcu_gp_seq_needed;/* Furthest future GP needed. */ +unsigned long srcu_gp_seq_needed_exp;/* Furthest future exp GP. */ bool srcu_cblist_invoking;/* Invoking these CBs? */ struct delayed_work work;/* Context for CB invoking. */ struct rcu_head srcu_barrier_head;/* For srcu_barrier() use. */ struct srcu_node *mynode;/* Leaf srcu_node. */ +unsigned long grpmask;/* Mask for leaf srcu_node */ +/* ->srcu_data_have_cbs[]. */ int cpu;struct srcu_struct *sp; };
@@ -59,6 +62,9 @@ struct srcu_node {unsigned long srcu_have_cbs[4 ];/* GP seq for children */ /* having CBs, but only */ /* is > ->srcu_gq_seq. */ +unsigned long srcu_data_have_cbs[4 ];/* Which srcu_data structs */ +/* have CBs for given GP? */ +unsigned long srcu_gp_seq_needed_exp;/* Furthest future exp GP. */ struct srcu_node *srcu_parent;/* Next up in tree. */ int grplo;/* Least CPU for node. */ int grphi;/* Biggest CPU for node. */
@@ -77,7 +83,8 @@ struct srcu_struct {unsigned int srcu_idx;/* Current rdr array element. */ unsigned long srcu_gp_seq;/* Grace-period seq #. */ unsigned long srcu_gp_seq_needed;/* Latest gp_seq needed. */ -atomic_t srcu_exp_cnt;/* # ongoing expedited GPs. */ +unsigned long srcu_gp_seq_needed_exp;/* Furthest future exp GP. */ +unsigned long srcu_last_gp_end;/* Last GP end timestamp (ns) */ struct srcu_data __percpu *sda;/* Per-CPU srcu_data array. */ unsigned long srcu_barrier_seq;/* srcu_barrier seq #. */ struct mutex srcu_barrier_mutex;/* Serialize barrier ops. */
@@ -136,4 +143,8 @@ void synchronize_srcu_expedited(struct srcu_struct *sp);void srcu_barrier (struct srcu_struct *sp);unsigned long srcu_batches_completed (struct srcu_struct *sp); +void srcutorture_get_gp_data (enum rcutorture_type test_type, +struct srcu_struct *sp,int *flags, +unsigned long *gpnum,unsigned long *completed); +#endif
kernel/rcu/rcutorture.c +5 -3
@@ -1360,12 +1360,14 @@ rcu_torture_stats_print(void) cur_ops->stats ();if (rtcv_snap == rcu_torture_current_version && rcu_torture_current !=NULL ) { -int __maybe_unused flags; -unsigned long __maybe_unused gpnum; -unsigned long __maybe_unused completed; +int __maybe_unused flags =0 ; +unsigned long __maybe_unused gpnum =0 ; +unsigned long __maybe_unused completed =0 ;rcutorture_get_gp_data (cur_ops->ttype, &flags, &gpnum, &completed); +srcutorture_get_gp_data (cur_ops->ttype, srcu_ctlp, + &flags, &gpnum, &completed); wtp =READ_ONCE (writer_task); pr_alert("??? Writer stall state %s(%d) g%lu c%lu f%#x ->state %#lx\n", rcu_torture_writer_state_getname(),
kernel/rcu/srcutree.c +200 -42
443 lines not on this page · raw message
kernel/rcu/tree.c +4 -8
@@ -704,15 +704,11 @@ void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags,default :break ; } -if (rsp !=NULL ) { - *flags =READ_ONCE (rsp->gp_flags); - *gpnum =READ_ONCE (rsp->gpnum); - *completed =READ_ONCE (rsp->completed); +if (rsp ==NULL )return ; - } - *flags =0 ; - *gpnum =0 ; - *completed =0 ; + *flags =READ_ONCE (rsp->gp_flags); + *gpnum =READ_ONCE (rsp->gpnum); + *completed =READ_ONCE (rsp->completed); }EXPORT_SYMBOL_GPL (rcutorture_get_gp_data);
Quoted 2 lines · Paul E. McKenney
And a sneak preview, semi-tested. If you get a chance to run this, please let me know now it goes.
That took 'time stress-cpu-hotplug.sh' down to 48s, close to classic.
-Mike
Quoted 6 lines · Mike Galbraith
Quoted 2 lines · Paul E. McKenney
And a sneak preview, semi-tested. If you get a chance to run this, please let me know now it goes.
That took 'time
stress-cpu-hotplug.sh' down to 48s, close to classic.
Woo-hoo!!! ;-)
And thank you for your testing efforts!
Should I be comparing this with the 55s number from your initial email, or to the 39s number?
Either way, given the unusual nature of Steven's hotplug stress test, I believe that I am good enough for this merge window. But if we are talking 48s for Tree SRCU vs. 39s with Classic SRCU, it would be good to at least understand where the remaining slowdown is. Here are a couple of possible causes:
o My holdoff is too long. I set it to 50 microseconds based
on your trace, which shows a minimum grace-period separation of 118 microseconds. But perhaps the trace was too short to show the full variation. One way to check this is to run with srcutree.exp_holdoff=25000 or some such. (Please note that srcutree.exp_holdoff is in nanoseconds, -not- microseconds.)
o My expedited throttling is too aggressive. This is controlled
by the following lines of code in srcu_gp_end() in the file kernel/rcu/srcutree.c: /* Throttle expedited grace periods: Should be rare! */ srcu_reschedule(sp, rcu_seq_ctr(gpseq) & 0x3ff ? 0 : SRCU_INTERVAL); The "0x3ff" says that one in 1024 grace periods should be forced to be at least partially non-expedited, regardless of anything else. If making this be (say) "0xfff" gets you three-quarters of the way to the 39s, that indicates that this is the controlling factor.
o Of course, another question is how much variation there is
in the timing of that stress test.
If further reduction is needed, and none of these help, could you please send me a trace of the full run of the same form as the last one you sent me, covering calls to and returns from call_srcu(), synchronize_srcu(), and synchronize_srcu_expedited()?
Thanx, paul
Quoted 6 lines · Paul E. McKenney
Quoted 5 lines · Mike Galbraith
Quoted 2 lines · Paul E. McKenney
And a sneak preview, semi-tested. If you get a chance to run this, please let me know now it goes.
Quoted 8 lines
Quoted 1 line
That took 'time
stress-cpu-hotplug.sh' down to 48s, close to classic.Woo-hoo!!! ;-)
And thank you for your testing efforts!
Should I be comparing this with the 55s number from your initial email, or to the 39s number?
Should be the 39, but I'll get you a new tree vs classic number in this same tree after I finish bisecting TSC going wonky.
-Mike
Quoted 11 lines · Mike Galbraith
Quoted 10 lines · Paul E. McKenney
Quoted 6 lines · Mike Galbraith
Quoted 2 lines · Paul E. McKenney
And a sneak preview, semi-tested. If you get a chance to run this, please let me know now it goes.
That took 'time
stress-cpu-hotplug.sh' down to 48s, close to classic.Woo-hoo!!! ;-)
Quoted 7 lines
Quoted 4 lines
And thank you for your testing efforts!
Should I be comparing this with the 55s number from your initial email, or to the 39s number?
Should be the 39, but I'll get you a new tree vs classic number in this same tree after I finish bisecting TSC going wonky.
Sounds good!!!
In theory, given 4-millisecond SRCU grace periods and 225,000 of them per run (estimated based on experience and on numbers you have already provided me), the 0x3ff should be worth only about one second of delay over the full test. On a really good day, I could probably invent some statistics to estimate the slowdown from the 50-microsecond cutoff, but today is not quite that good a day. And besides, the only difference between theory and practice is that in theory they are both the same. ;-)
So I look forward to seeing your measurements! And best of everything with the wonky TSC!!!
Thanx, Paul
Quoted 6 lines · Mike Galbraith
Quoted 2 lines · Paul E. McKenney
Should I be comparing this with the 55s number from your initial email, or to the 39s number?
Should be the 39...
And 39 it is.
-Mike
Quoted 9 lines · Mike Galbraith
Quoted 6 lines · Mike Galbraith
Quoted 2 lines · Paul E. McKenney
Should I be comparing this with the 55s number from your initial email, or to the 39s number?
Should be the 39...
And 39 it is.
Thank you again for your testing efforts!
OK, I do need to do more work. My current guess is that I should have set the default for srcutree.exp_holdoff to 25*1000 instead of 50*1000. But I am sure that further data will show me the error of my ways. ;-)
Thanx, Paul
Quoted 9 lines · Paul E. McKenney
Quoted 8 lines · Mike Galbraith
Quoted 6 lines · Mike Galbraith
Quoted 2 lines · Paul E. McKenney
Should I be comparing this with the 55s number from your initial email, or to the 39s number?
Should be the 39...
Quoted 7 lines
Quoted 1 line
And 39 it is.
Thank you again for your testing efforts!
OK, I do need to do more work. My current guess is that I should have set the default for srcutree.exp_holdoff to 25*1000 instead of 50*1000. But I am sure that further data will show me the error of my ways. ;-)
Oh, and my I add your Tested-by?
Thanx, Paul
Quoted 3 lines · Paul E. McKenney
Quoted 3 lines
OK, I do need to do more work. My current guess is that I should have set the default for srcutree.exp_holdoff to 25*1000 instead of 50*1000. But I am sure that further data will show me the error of my ways. ;-)
I can give that a go tomorrow (too crispy now).
Quoted 1 line
Oh, and my I add your Tested-by?
Sure.
-Mike
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Paul E. McKenney
Quoted 3 lines
OK, I do need to do more work. My current guess is that I should have set the default for srcutree.exp_holdoff to 25*1000 instead of 50*1000. But I am sure that further data will show me the error of my ways. ;-)
I can give that a go tomorrow (too crispy now).
Sounds good! I think that validating a 75x reduction is good for today. Get some rest!
Quoted 3 lines
Quoted 1 line
Oh, and my I add your Tested-by?
Sure.
Done, thank you!
Thanx, Paul
Quoted 7 lines · Mike Galbraith
Quoted 3 lines · Paul E. McKenney
Quoted 3 lines
OK, I do need to do more work. My current guess is that I should have set the default for srcutree.exp_holdoff to 25*1000 instead of 50*1000. But I am sure that further data will show me the error of my ways. ;-)
I can give that a go tomorrow (too crispy now).
real 0m39.177s user 0m0.225s sys 0m5.344s
tadaaa (tricorder noises... it's dead Jim)
Quoted 7 lines · Mike Galbraith
Quoted 6 lines · Mike Galbraith
Quoted 3 lines · Paul E. McKenney
Quoted 3 lines
OK, I do need to do more work. My current guess is that I should have set the default for srcutree.exp_holdoff to 25*1000 instead of 50*1000. But I am sure that further data will show me the error of my ways. ;-)
Quoted 7 lines
Quoted 1 line
I can give that a go tomorrow (too crispy now).
real 0m39.177s user 0m0.225s sys 0m5.344stadaaa (tricorder noises... it's dead Jim)
This is with srcutree.exp_holdoff set to 25*1000? If so, great news! Really easy to adjust the default. ;-)
Thanx, Paul
Quoted 1 line · Paul E. McKenney
This is with srcutree.exp_holdoff set to 25*1000?
Yup.
Quoted 5 lines · Mike Galbraith
Quoted 1 line · Paul E. McKenney
This is with srcutree.exp_holdoff set to 25*1000?
Yup.
And please see below for the trivial patch, just for confirmation. May I add your Tested-by?
Thanx, Paul
------------------------------------------------------------------------
kernel/rcu/srcutree.c +1 -1
@@ -39,7 +39,7 @@#include "rcu.h" -ulong exp_holdoff =50 *1000 ;/* Holdoff (ns) for auto-expediting. */ +ulong exp_holdoff =25 *1000 ;/* Holdoff (ns) for auto-expediting. */ module_param (exp_holdoff, ulong,0444 );static void srcu_invoke_callbacks (struct work_struct *work);
Quoted 9 lines · Paul E. McKenney
Quoted 5 lines · Mike Galbraith
Quoted 1 line · Paul E. McKenney
This is with srcutree.exp_holdoff set to 25*1000?
Yup.
And please see below for the trivial patch, just for confirmation. May I add your Tested-by?
Sure.
kernel/rcu/srcutree.c line 42
Quoted 10 lines
kernel/rcu/srcutree.c +1
-1@@ -39,7 +39,7 @@#include "rcu.h" -ulong exp_holdoff =50 *1000 ;/* Holdoff (ns) for auto-expediting. */ +ulong exp_holdoff =25 *1000 ;/* Holdoff (ns) for auto-expediting. */
Yup, exactly what I did.
Quoted 19 lines · Mike Galbraith
Quoted 9 lines · Paul E. McKenney
Quoted 5 lines · Mike Galbraith
Quoted 1 line · Paul E. McKenney
This is with srcutree.exp_holdoff set to 25*1000?
Yup.
And please see below for the trivial patch, just for confirmation. May I add your Tested-by?
Sure.
Quoted 6 lines
kernel/rcu/srcutree.c +0
-0@@ -39,7 +39,7 @@
Quoted 6 lines
kernel/rcu/srcutree.c line 42
Quoted 4 lines
kernel/rcu/srcutree.c +1
-1#include "rcu.h" -ulong exp_holdoff =50 *1000 ;/* Holdoff (ns) for auto-expediting. */ +ulong exp_holdoff =25 *1000 ;/* Holdoff (ns) for auto-expediting. */ Yup, exactly what I did.
Woo-hoo!!!
I will beat up on this a bit and send a pull request to Ingo.
And thank you very much for finding this, gathering the trace data (which was critically important to my figuring out what was going on), and for testing the patches!
Thanx, Paul