psi: pressure stall information for CPU, memory, and IO v4
Subsystems (27)
Johannes Weiner
maintainercover psi: pressure stall information for CPU, memory, and IO v4
This version 4 of the PSI series incorporates feedback from Peter and fixes two races in the lockless aggregator that Suren found in his testing and which caused the sample calculation to sometimes underflow and record bogusly large samples; details at the bottom of this email.
Overview
PSI reports the overall wallclock time in which the tasks in a system (or cgroup) wait for (contended) hardware resources.
This helps users understand the resource pressure their workloads are under, which allows them to rootcause and fix throughput and latency problems caused by overcommitting, underprovisioning, suboptimal job placement in a grid; as well as anticipate major disruptions like OOM.
Real-world applications
We're using the data collected by PSI (and its previous incarnation, memdelay) quite extensively at Facebook, and with several success stories.
One usecase is avoiding OOM hangs/livelocks. The reason these happen is because the OOM killer is triggered by reclaim not being able to free pages, but with fast flash devices there is *always* some clean and uptodate cache to reclaim; the OOM killer never kicks in, even as tasks spend 90% of the time thrashing the cache pages of their own executables. There is no situation where this ever makes sense in practice. We wrote a <100 line POC python script to monitor memory pressure and kill stuff way before such pathological thrashing leads to full system losses that would require forcible hard resets.
We've since extended and deployed this code into other places to guarantee latency and throughput SLAs, since they're usually violated way before the kernel OOM killer would ever kick in.
It is available here: https://github.com/facebookincubator/oomd
Eventually we probably want to trigger the in-kernel OOM killer based on extreme sustained pressure as well, so that Linux can avoid memory livelocks - which technically aren't deadlocks, but to the user indistinguishable from them - out of the box. We'd continue using OOMD as the first line of defense to ensure workload health and implement complex kill policies that are beyond the scope of the kernel.
We also use PSI memory pressure for loadshedding. Our batch job infrastructure used to use heuristics based on various VM stats to anticipate OOM situations, with lackluster success. We switched it to PSI and managed to anticipate and avoid OOM kills and lockups fairly reliably. The reduction of OOM outages in the worker pool raised the pool's aggregate productivity, and we were able to switch that service to smaller machines.
Lastly, we use cgroups to isolate a machine's main workload from maintenance crap like package upgrades, logging, configuration, as well as to prevent multiple workloads on a machine from stepping on each others' toes. We were not able to configure this properly without the pressure metrics; we would see latency or bandwidth drops, but it would often be hard to impossible to rootcause it post-mortem.
We now log and graph pressure for the containers in our fleet and can trivially link latency spikes and throughput drops to shortages of specific resources after the fact, and fix the job config/scheduling.
PSI has also received testing, feedback, and feature requests from Android and EndlessOS for the purpose of low-latency OOM killing, to intervene in pressure situations before the UI starts hanging.
How do you use this feature?
A kernel with CONFIG_PSI=y will create a /proc/pressure directory with 3 files: cpu, memory, and io. If using cgroup2, cgroups will also have cpu.pressure, memory.pressure and io.pressure files, which simply aggregate task stalls at the cgroup level instead of system-wide.
The cpu file contains one line:
some avg10=2.04 avg60=0.75 avg300=0.40 total=157656722
The averages give the percentage of walltime in which one or more tasks are delayed on the runqueue while another task has the CPU. They're recent averages over 10s, 1m, 5m windows, so you can tell short term trends from long term ones, similarly to the load average.
The total= value gives the absolute stall time in microseconds. This allows detecting latency spikes that might be too short to sway the running averages. It also allows custom time averaging in case the 10s/1m/5m windows aren't adequate for the usecase (or are too coarse with future hardware).
What to make of this "some" metric? If CPU utilization is at 100% and CPU pressure is 0, it means the system is perfectly utilized, with one runnable thread per CPU and nobody waiting. At two or more runnable tasks per CPU, the system is 100% overcommitted and the pressure average will indicate as much. From a utilization perspective this is a great state of course: no CPU cycles are being wasted, even when 50% of the threads were to go idle (as most workloads do vary). From the perspective of the individual job it's not great, however, and they would do better with more resources. Depending on what your priority and options are, raised "some" numbers may or may not require action.
The memory file contains two lines:
some avg10=70.24 avg60=68.52 avg300=69.91 total=3559632828 full avg10=57.59 avg60=58.06 avg300=60.38 total=3300487258
The some line is the same as for cpu, the time in which at least one task is stalled on the resource. In the case of memory, this includes waiting on swap-in, page cache refaults and page reclaim.
The full line, however, indicates time in which *nobody* is using the CPU productively due to pressure: all non-idle tasks are waiting for memory in one form or another. Significant time spent in there is a good trigger for killing things, moving jobs to other machines, or dropping incoming requests, since neither the jobs nor the machine overall are making too much headway.
The io file is similar to memory. Because the block layer doesn't have a concept of hardware contention right now (how much longer is my IO request taking due to other tasks?), it reports CPU potential lost on all IO delays, not just the potential lost due to competition.
FAQ
Q: How is PSI's CPU component different from the load average?
A: There are several quirks in the load average that make it hard to
impossible to tell how overcommitted the CPU really is.
1. The load average is reported as a raw number of active tasks. You need to know how many CPUs there are in the system, how many CPUs the workload is allowed to use, then think about what the proportion between load and the number of CPUs mean for the tasks trying to run.
PSI reports the percentage of wallclock time in which tasks are waiting for a CPU to run on. It doesn't matter how many CPUs are present or usable. The number always tells the quality of life of tasks in the system or in a particular cgroup.
2. The shortest averaging window is 1m, which is extremely coarse, and it's sampled in 5s intervals. A *lot* can happen on a CPU in 5 seconds. This *may* be able to identify persistent long-term trends and very clear and obvious overloads, but it's unusable for latency spikes and more subtle overutilization.
PSI's shortest window is 10s. It also exports the cumulative stall times (in microseconds) of synchronously recorded events.
3. On Linux, the load average for historical reasons includes all TASK_UNINTERRUPTIBLE tasks. This gives a broader sense of how busy the system is, but on the flipside it doesn't distinguish whether tasks are likely to contend over the CPU or IO - which obviously requires very different interventions from a sys admin or a job scheduler.
PSI reports independent metrics for CPU and IO. You can tell which resource is making the tasks wait, but in conjunction still see how overloaded the system is overall.
Q: What's the cost / performance impact of this feature?
A: PSI's primary cost is in the scheduler, in particular task wakeups
and sleeps.
I benchmarked this code using Facebook's two most scheduling sensitive workloads: memcache and webserver. They handle a ton of small requests - lots of wakeups and sleeps with little actual work in between - so they tend to be canaries for scheduler regressions.
In the tests, the boxes were handling live traffic over the course of several hours. Half the machines, the control, ran with CONFIG_PSI=n.
For memcache I used eight machines total. They're 2-socket, 14 core, 56 thread boxes. The test runs for half the test period, flips the test and control kernels on the hardware to rule out HW factors, DC location etc., then runs the other half of the test.
For the webservers, I used 32 machines total. They're single socket, 16 core, 32 thread machines.
During the memcache test, CPU load was nopsi=78.05% psi=78.98% in the first half and nopsi=77.52% psi=78.25%, so PSI added between 0.7 and 0.9 percentage points to the CPU load, a difference of about 1%.
UPDATE: I re-ran this test with the v3 version of this patch set and the CPU utilization was equivalent between test and control.
UPDATE: v4 is on par with v3.
As far as end-to-end request latency from the client perspective goes, we don't sample those finely enough to capture the requests going to those particular machines during the test, but we know the p50 turnaround time in this workload is 54us, and perf bench sched pipe on those machines show nopsi=5.232666 us/op and psi=5.587347 us/op, so this doesn't add much here either.
The profile for the pipe benchmark shows:
0.87% sched-pipe [kernel.vmlinux] [k] psi_group_change
0.83% perf.real [kernel.vmlinux] [k] psi_group_change
0.82% perf.real [kernel.vmlinux] [k] psi_task_change
0.58% sched-pipe [kernel.vmlinux] [k] psi_task_changeThe webserver load is running inside 4 nested cgroup levels. The CPU load with both nopsi and psi kernels was indistinguishable at 81%.
For comparison, we had to disable the cgroup cpu controller on the webservers because it added 4 percentage points to the CPU% during this same exact test.
Versions of this accounting code now run on 80% of our fleet. None of our workloads have reported regressions during the rollout.
These patches are against v4.18. They're maintained against upstream here as well: http://git.cmpxchg.org/cgit.cgi/linux-psi.git
Documentation/accounting/psi.txt | 73 +++ Documentation/admin-guide/cgroup-v2.rst | 18 + arch/powerpc/platforms/cell/cpufreq_spudemand.c | 2 +- arch/powerpc/platforms/cell/spufs/sched.c | 9 +- arch/s390/appldata/appldata_os.c | 4 - drivers/cpuidle/governors/menu.c | 4 - fs/proc/loadavg.c | 3 - include/linux/cgroup-defs.h | 4 + include/linux/cgroup.h | 15 + include/linux/delayacct.h | 23 + include/linux/mmzone.h | 1 + include/linux/page-flags.h | 5 +- include/linux/psi.h | 53 ++ include/linux/psi_types.h | 92 +++ include/linux/sched.h | 10 + include/linux/sched/loadavg.h | 24 +- include/linux/swap.h | 2 +- include/trace/events/mmflags.h | 1 + include/uapi/linux/taskstats.h | 6 +- init/Kconfig | 19 + kernel/cgroup/cgroup.c | 45 +- kernel/debug/kdb/kdb_main.c | 7 +- kernel/delayacct.c | 15 + kernel/fork.c | 4 + kernel/sched/Makefile | 1 + kernel/sched/core.c | 16 +- kernel/sched/loadavg.c | 139 ++-- kernel/sched/psi.c | 752 ++++++++++++++++++++++ kernel/sched/sched.h | 178 ++--- kernel/sched/stats.h | 86 +++ mm/compaction.c | 5 + mm/filemap.c | 27 +- mm/huge_memory.c | 1 + mm/memcontrol.c | 2 + mm/migrate.c | 2 + mm/page_alloc.c | 9 + mm/swap_state.c | 1 + mm/vmscan.c | 10 + mm/vmstat.c | 1 + mm/workingset.c | 113 ++-- tools/accounting/getdelays.c | 8 +- 41 files changed, 1543 insertions(+), 247 deletions(-)
Changes in v2:
- Extensive documentation and comment update. Per everybody. In particular, I've added a much more detailed explanation of the SMP model, which caused some misunderstandings last time.
- Uninlined calc_load_n(), as it was just too fat. Per Peter.
- Split kernel/sched/stats.h churn into its own commit to avoid noise in the main patch and explain the reshuffle. Per Peter.
- Abstracted this_rq_lock_irq(). Per Peter.
- Eliminated cumulative clock drift error. Per Peter.
- Packed the per-cpu datastructure. Per Peter.
- Fixed 64-bit divisions on 32 bit. Per Peter.
- Added outer-most psi_disabled checks. Per Peter.
- Fixed some coding style issues. Per Peter.
- Fixed a bug in the lazy clock. Per Suren.
- On-demand stat aggregation when user reads. Per Suren.
- Fixed task state corruption on preemption race. Per Suren.
- Fixed a CONFIG_PSI=n build error.
- Minor cleanups, optimizations.
Changes in v3:
- Packed scheduler hotpath data into one cacheline, as per Peter and Linus
- Implemented live state aggregation without the rq lock, as per Peter
- do_div -> div64_ul and some other cleanups, as per Peter
- Dropped unnecessary SCHED_INFO dependency, as per Peter
- Realtime sampling period and slipped sample handling, as per Tejun
- Fixed 64-bit divsion on 32 bit & checkpatch warnings, as per Andrew
Changes in v4:
- Fixed an unsafe cpu_curr() dereference from the live aggregator. This was there to detect active reclaimers on a CPU. Instead of adding an expensive task switching callback, sample that state from scheduler_tick(). As per Peter.
- Use for_each_possible_cpu() instead of the online mask when aggregating per-cpu samples, to avoid rare artifacts from CPU hotplugging. As per Peter
- Refactor the aggregation loop to be more explicit about extracting nonidle time - the coefficient for all other state times - first. As per Peter.
- Fixed a race condition between the scheduler and the live aggregator in which the aggregator misses a previously observed live state that is no longer live but hasn't made it into the recorded time bucket yet. In this case the 'times - times_prev' sampling will underflow and cause us to record a bogusly large time sample. This isn't fixable with memory barriers, since we also need to avoid seeing the delta simultaneously in the live state and in the recorded time buckets. Added a seqcount to ensure a coherent view from the aggregator. As per Suren.
- Fixed a related problem where the clock of the state change (rq_clock) is behind that of the aggregator (cpu_clock). A race between the two can cause the aggregator to observe a longer live state time than what the scheduler ends up recording - again leading to the same delta detection underflow and bogus sample recording. The state changer has to use cpu_clock from within the seqcount section. As per Suren.
- Note that these changes didn't affect the memcache benchmark results.
Johannes Weiner
maintainerPATCH 1/9 mm: workingset: don't drop refault information prematurely
From: Johannes Weiner <[email protected]>
If we keep just enough refault information to match the *current* page cache during reclaim time, we could lose a lot of events when there is only a temporary spike in non-cache memory consumption that pushes out all the cache. Once cache comes back, we won't see those refaults. They might not be actionable for LRU aging, but we want to know about them for measuring memory pressure.
--- mm/workingset.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
mm/workingset.c +9 -9
@@ -364,7 +364,7 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker, {unsigned long max_nodes;unsigned long nodes; -unsigned long cache; +unsigned long pages;/* list_lru lock nests inside the IRQ-safe i_pages lock */ local_irq_disable ();
@@ -393,14 +393,14 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker,* * PAGE_SIZE / radix_tree_nodes / node_entries * 8 / PAGE_SIZE */ -if (sc->memcg) { - cache =mem_cgroup_node_nr_lru_pages (sc->memcg, sc->nid, -LRU_ALL_FILE ); - }else { - cache =node_page_state (NODE_DATA (sc->nid),NR_ACTIVE_FILE ) + -node_page_state (NODE_DATA (sc->nid),NR_INACTIVE_FILE ); - } - max_nodes = cache >> (RADIX_TREE_MAP_SHIFT -3 ); +#ifdef CONFIG_MEMCG +if (sc->memcg) + pages =page_counter_read (&sc->memcg->memory); + else +#endif + pages =node_present_pages (sc->nid); + + max_nodes = pages >> (RADIX_TREE_MAP_SHIFT -3 );if (nodes <= max_nodes)return 0 ;
Signature · 1 line
2.18.0
Johannes Weiner
maintainerPATCH 2/9 mm: workingset: tell cache transitions from workingset thrashing
In reply to Johannes WeinerRefaults happen during transitions between workingsets as well as in-place thrashing. Knowing the difference between the two has a range of applications, including measuring the impact of memory shortage on the system performance, as well as the ability to smarter balance pressure between the filesystem cache and the swap-backed workingset.
During workingset transitions, inactive cache refaults and pushes out established active cache. When that active cache isn't stale, however, and also ends up refaulting, that's bonafide thrashing.
Introduce a new page flag that tells on eviction whether the page has been active or not in its lifetime. This bit is then stored in the shadow entry, to classify refaults as transitioning or thrashing.
How many page->flags does this leave us with on 32-bit?
20 bits are always page flags
21 if you have an MMU
23 with the zone bits for DMA, Normal, HighMem, Movable
29 with the sparsemem section bits
30 if PAE is enabled
31 with this patch.
So on 32-bit PAE, that leaves 1 bit for distinguishing two NUMA nodes. If that's not enough, the system can switch to discontigmem and re-gain the 6 or 7 sparsemem section bits.
v4:
- fix a typo in the comments, as per Suren
--- include/linux/mmzone.h | 1 + include/linux/page-flags.h | 5 +- include/linux/swap.h | 2 +- include/trace/events/mmflags.h | 1 + mm/filemap.c | 9 ++-- mm/huge_memory.c | 1 + mm/memcontrol.c | 2 + mm/migrate.c | 2 + mm/swap_state.c | 1 + mm/vmscan.c | 1 + mm/vmstat.c | 1 + mm/workingset.c | 95 ++++++++++++++++++++++------------ 12 files changed, 79 insertions(+), 42 deletions(-)
include/linux/mmzone.h +1 -0
@@ -163,6 +163,7 @@ enum node_stat_item {NR_ISOLATED_FILE ,/* Temporary isolated pages from file lru */ WORKINGSET_REFAULT ,WORKINGSET_ACTIVATE , +WORKINGSET_RESTORE ,WORKINGSET_NODERECLAIM ,NR_ANON_MAPPED ,/* Mapped anonymous pages */ NR_FILE_MAPPED, /* pagecache pages mapped into pagetables.
include/linux/page-flags.h +4 -1
@@ -69,13 +69,14 @@*/ enum pageflags { PG_locked,/* Page is locked. Don't touch. */ - PG_error, PG_referenced, PG_uptodate, PG_dirty, PG_lru, PG_active, + PG_workingset, PG_waiters,/* Page has waiters, check its waitqueue. Must be bit #7 and in the same byte as "PG_locked" */ + PG_error, PG_slab, PG_owner_priv_1,/* Owner use. If pagecache, fs may use*/ PG_arch_1,
@@ -280,6 +281,8 @@ PAGEFLAG(Dirty, dirty, PF_HEAD) TESTSCFLAG(Dirty, dirty, PF_HEAD) PAGEFLAG(LRU, lru, PF_HEAD) __CLEARPAGEFLAG(LRU, lru, PF_HEAD) PAGEFLAG(Active, active, PF_HEAD) __CLEARPAGEFLAG(Active, active, PF_HEAD) TESTCLEARFLAG(Active, active, PF_HEAD) +PAGEFLAG(Workingset, workingset, PF_HEAD) + TESTCLEARFLAG(Workingset, workingset, PF_HEAD) __PAGEFLAG(Slab, slab, PF_NO_TAIL) __PAGEFLAG(SlobFree, slob_free, PF_NO_TAIL)PAGEFLAG (Checked, checked,PF_NO_COMPOUND )/* Used by some filesystems */
include/linux/swap.h +1 -1
@@ -296,7 +296,7 @@ struct vma_swap_readahead {/* linux/mm/workingset.c */ void *workingset_eviction (struct address_space *mapping,struct page *page); -bool workingset_refault (void *shadow); +void workingset_refault (struct page *page,void *shadow);void workingset_activation (struct page *page);/* Do not use directly, use workingset_lookup_update */
include/trace/events/mmflags.h +1 -0
@@ -88,6 +88,7 @@ {1UL << PG_dirty, "dirty" }, \ {1UL << PG_lru, "lru" }, \ {1UL << PG_active, "active" }, \ + {1UL << PG_workingset, "workingset" }, \ {1UL << PG_slab, "slab" }, \ {1UL << PG_owner_priv_1, "owner_priv_1" }, \ {1UL << PG_arch_1, "arch_1" }, \
mm/filemap.c +3 -6
@@ -915,12 +915,9 @@ int add_to_page_cache_lru(struct page *page, struct address_space *mapping,* data from the working set, only to cache data that will * get overwritten with something else, is a waste of memory. */ -if (!(gfp_mask & __GFP_WRITE) && - shadow &&workingset_refault (shadow)) { -SetPageActive (page); -workingset_activation (page); - }else -ClearPageActive (page); +WARN_ON_ONCE (PageActive (page)); +if (!(gfp_mask & __GFP_WRITE) && shadow) +workingset_refault (page, shadow);lru_cache_add (page); }return ret;
mm/huge_memory.c +1 -0
@@ -2369,6 +2369,7 @@ static void __split_huge_page_tail(struct page *head, int tail, (1L << PG_mlocked) | (1L << PG_uptodate) | (1L << PG_active) | + (1L << PG_workingset) | (1L << PG_locked) | (1L << PG_unevictable) | (1L << PG_dirty)));
mm/memcontrol.c +2 -0
@@ -5329,6 +5329,8 @@ static int memory_stat_show(struct seq_file *m, void *v) stat[WORKINGSET_REFAULT]);seq_printf (m,"workingset_activate %lu\n" , stat[WORKINGSET_ACTIVATE ]); +seq_printf (m,"workingset_restore %lu\n" , + stat[WORKINGSET_RESTORE ]);seq_printf (m,"workingset_nodereclaim %lu\n" , stat[WORKINGSET_NODERECLAIM ]);
mm/migrate.c +2 -0
@@ -682,6 +682,8 @@ void migrate_page_states(struct page *newpage, struct page *page)SetPageActive (newpage); }else if (TestClearPageUnevictable (page))SetPageUnevictable (newpage); +if (PageWorkingset (page)) +SetPageWorkingset (newpage);if (PageChecked (page))SetPageChecked (newpage);if (PageMappedToDisk (page))
mm/swap_state.c +1 -0
@@ -448,6 +448,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,/* * Initiate read into locked page and return. */ +SetPageWorkingset (new_page);lru_cache_add_anon (new_page); *new_page_allocated = true;return new_page;
mm/vmscan.c +1 -0
@@ -1976,6 +1976,7 @@ static void shrink_active_list(unsigned long nr_to_scan, }ClearPageActive (page);/* we are de-activating */ +SetPageWorkingset (page);list_add (&page->lru, &l_inactive); }
mm/vmstat.c +1 -0
@@ -1145,6 +1145,7 @@ const char * const vmstat_text[] = {"nr_isolated_file" ,"workingset_refault" ,"workingset_activate" , +"workingset_restore" ,"workingset_nodereclaim" ,"nr_anon_pages" , "nr_mapped",
mm/workingset.c +61 -34
196 lines not on this page · raw message
Signature · 1 line
2.18.0
Johannes Weiner
maintainerPATCH 3/9 delayacct: track delays from thrashing cache pages
In reply to Johannes WeinerDelay accounting already measures the time a task spends in direct reclaim and waiting for swapin, but in low memory situations tasks spend can spend a significant amount of their time waiting on thrashing page cache. This isn't tracked right now.
To know the full impact of memory contention on an individual task, measure the delay when waiting for a recently evicted active cache page to read back into memory.
Also update tools/accounting/getdelays.c:
[hannes@computer accounting]$ sudo ./getdelays -d -p 1 print delayacct stats ON
PID 1
CPU count real total virtual total delay total delay average
50318 745000000 847346785 400533713 0.008ms
IO count delay total delay average
435 122601218 0ms
SWAP count delay total delay average
0 0 0ms
RECLAIM count delay total delay average
0 0 0ms
THRASHING count delay total delay average
19 12621439 0ms
--- include/linux/delayacct.h | 23 +++++++++++++++++++++++ include/uapi/linux/taskstats.h | 6 +++++- kernel/delayacct.c | 15 +++++++++++++++ mm/filemap.c | 11 +++++++++++ tools/accounting/getdelays.c | 8 +++++++- 5 files changed, 61 insertions(+), 2 deletions(-)
include/linux/delayacct.h +23 -0
@@ -57,7 +57,12 @@ struct task_delay_info {u64 freepages_start;u64 freepages_delay;/* wait for memory reclaim */ + +u64 thrashing_start; +u64 thrashing_delay;/* wait for thrashing page */ +u32 freepages_count;/* total count of memory reclaim */ + u32 thrashing_count; /* total count of thrash waits */ };#endif
@@ -76,6 +81,8 @@ extern int __delayacct_add_tsk(struct taskstats *, struct task_struct *);extern __u64 __delayacct_blkio_ticks (struct task_struct *);extern void __delayacct_freepages_start (void );extern void __delayacct_freepages_end (void ); +extern void __delayacct_thrashing_start (void ); +extern void __delayacct_thrashing_end (void );static inline int delayacct_is_task_waiting_on_io (struct task_struct *p) {
@@ -156,6 +163,18 @@ static inline void delayacct_freepages_end(void) __delayacct_freepages_end(); } +static inline void delayacct_thrashing_start (void ) +{ +if (current->delays) +__delayacct_thrashing_start (); +} + +static inline void delayacct_thrashing_end (void ) +{ +if (current->delays) +__delayacct_thrashing_end (); +} +#else static inline void delayacct_set_flag (int flag) {}
@@ -182,6 +201,10 @@ static inline void delayacct_freepages_start(void) {}static inline void delayacct_freepages_end (void ) {} +static inline void delayacct_thrashing_start (void ) +{} +static inline void delayacct_thrashing_end (void ) +{}#endif /* CONFIG_TASK_DELAY_ACCT */
include/uapi/linux/taskstats.h +5 -1
@@ -34,7 +34,7 @@*/ -#define TASKSTATS_VERSION 8 +#define TASKSTATS_VERSION 9#define TS_COMM_LEN 32/* should be >= TASK_COMM_LEN * in linux/sched.h */
@@ -164,6 +164,10 @@ struct taskstats {/* Delay waiting for memory reclaim */ __u64 freepages_count;__u64 freepages_delay_total; + +/* Delay waiting for thrashing page */ +__u64 thrashing_count; +__u64 thrashing_delay_total; };
kernel/delayacct.c +15 -0
@@ -135,9 +135,12 @@ int __delayacct_add_tsk(struct taskstats *d, struct task_struct *tsk) d->swapin_delay_total = (tmp < d->swapin_delay_total) ?0 : tmp; tmp = d->freepages_delay_total + tsk->delays->freepages_delay; d->freepages_delay_total = (tmp < d->freepages_delay_total) ?0 : tmp; + tmp = d->thrashing_delay_total + tsk->delays->thrashing_delay; + d->thrashing_delay_total = (tmp < d->thrashing_delay_total) ?0 : tmp; d->blkio_count += tsk->delays->blkio_count; d->swapin_count += tsk->delays->swapin_count; d->freepages_count += tsk->delays->freepages_count; + d->thrashing_count += tsk->delays->thrashing_count;raw_spin_unlock_irqrestore (&tsk->delays->lock, flags);return 0 ;
@@ -169,3 +172,15 @@ void __delayacct_freepages_end(void) ¤t->delays->freepages_count); } +void __delayacct_thrashing_start (void ) +{ + current->delays->thrashing_start =ktime_get_ns (); +} + +void __delayacct_thrashing_end (void ) +{ +delayacct_end (¤t->delays->lock, + ¤t->delays->thrashing_start, + ¤t->delays->thrashing_delay, + ¤t->delays->thrashing_count); +}
mm/filemap.c +11 -0
@@ -36,6 +36,7 @@#include <linux/cleancache.h> #include <linux/shmem_fs.h> #include <linux/rmap.h> +#include <linux/delayacct.h> #include "internal.h" #define CREATE_TRACE_POINTS
@@ -1073,8 +1074,15 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, {struct wait_page_queue wait_page;wait_queue_entry_t *wait = &wait_page.wait; +bool thrashing = false;int ret =0 ; +if (bit_nr == PG_locked && !PageSwapBacked (page) && + !PageUptodate (page) &&PageWorkingset (page)) { +delayacct_thrashing_start (); + thrashing = true; + } +init_wait (wait); wait->flags = lock ?WQ_FLAG_EXCLUSIVE :0 ; wait->func = wake_page_function;
@@ -1113,6 +1121,9 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q,finish_wait (q, wait); +if (thrashing) +delayacct_thrashing_end (); +/* * A signal could leave PageWaiters set. Clearing it here if * !waitqueue_active would be possible (by open-coding finish_wait),
tools/accounting/getdelays.c +7 -1
@@ -203,6 +203,8 @@ static void print_delayacct(struct taskstats *t)"SWAP %15s%15s%15s\n" " %15llu%15llu%15llums\n" "RECLAIM %12s%15s%15s\n" +" %15llu%15llu%15llums\n" +"THRASHING%12s%15s%15s\n" " %15llu%15llu%15llums\n" ,"count" ,"real total" ,"virtual total" , "delay total", "delay average",
@@ -222,7 +224,11 @@ static void print_delayacct(struct taskstats *t)"count" ,"delay total" ,"delay average" , (unsigned long long )t->freepages_count, (unsigned long long )t->freepages_delay_total, - average_ms(t->freepages_delay_total, t->freepages_count)); +average_ms (t->freepages_delay_total, t->freepages_count), +"count" ,"delay total" ,"delay average" , + (unsigned long long )t->thrashing_count, + (unsigned long long )t->thrashing_delay_total, + average_ms(t->thrashing_delay_total, t->thrashing_count)); }static void task_context_switch_counts (struct taskstats *t)
Signature · 1 line
2.18.0
Johannes Weiner
maintainerPATCH 4/9 sched: loadavg: consolidate LOAD_INT, LOAD_FRAC, CALC_LOAD
In reply to Johannes WeinerThere are several definitions of those functions/macros in places that mess with fixed-point load averages. Provide an official version.
--- .../platforms/cell/cpufreq_spudemand.c | 2 +- arch/powerpc/platforms/cell/spufs/sched.c | 9 +++----- arch/s390/appldata/appldata_os.c | 4 ---- drivers/cpuidle/governors/menu.c | 4 ---- fs/proc/loadavg.c | 3 --- include/linux/sched/loadavg.h | 21 +++++++++++++++---- kernel/debug/kdb/kdb_main.c | 7 +------ kernel/sched/loadavg.c | 15 ------------- 8 files changed, 22 insertions(+), 43 deletions(-)
arch/powerpc/platforms/cell/cpufreq_spudemand.c +1 -1
@@ -49,7 +49,7 @@ static int calc_freq(struct spu_gov_info_struct *info) cpu = info->policy->cpu; busy_spus =atomic_read (&cbe_spu_info[cpu_to_node (cpu)].busy_spus); -CALC_LOAD (info->busy_spus,EXP , busy_spus *FIXED_1 ); + info->busy_spus =calc_load (info->busy_spus,EXP , busy_spus *FIXED_1 );pr_debug ("cpu %d: busy_spus=%d, info->busy_spus=%ld\n" , cpu, busy_spus, info->busy_spus);
arch/powerpc/platforms/cell/spufs/sched.c +3 -6
@@ -987,9 +987,9 @@ static void spu_calc_load(void)unsigned long active_tasks;/* fixed-point */ active_tasks =count_active_contexts () *FIXED_1 ; -CALC_LOAD (spu_avenrun[0 ],EXP_1 , active_tasks); -CALC_LOAD (spu_avenrun[1 ],EXP_5 , active_tasks); -CALC_LOAD (spu_avenrun[2 ],EXP_15 , active_tasks); + spu_avenrun[0 ] =calc_load (spu_avenrun[0 ],EXP_1 , active_tasks); + spu_avenrun[1 ] =calc_load (spu_avenrun[1 ],EXP_5 , active_tasks); + spu_avenrun[2 ] =calc_load (spu_avenrun[2 ],EXP_15 , active_tasks); }static void spusched_wake (struct timer_list *unused)
@@ -1071,9 +1071,6 @@ void spuctx_switch_state(struct spu_context *ctx, } } -#define LOAD_INT (x) ((x) >> FSHIFT) -#define LOAD_FRAC (x) LOAD_INT(((x) & (FIXED_1-1)) * 100) -static int show_spu_loadavg (struct seq_file *s,void *private) {int a, b, c;
arch/s390/appldata/appldata_os.c +0 -4
@@ -25,10 +25,6 @@#include "appldata.h" - -#define LOAD_INT (x) ((x) >> FSHIFT) -#define LOAD_FRAC (x) LOAD_INT(((x) & (FIXED_1-1)) * 100) -/* * OS data *
drivers/cpuidle/governors/menu.c +0 -4
@@ -131,10 +131,6 @@ struct menu_device {int interval_ptr; }; - -#define LOAD_INT (x) ((x) >> FSHIFT) -#define LOAD_FRAC (x) LOAD_INT(((x) & (FIXED_1-1)) * 100) -static inline int get_loadavg (unsigned long load) {return LOAD_INT (load) *10 +LOAD_FRAC (load) /10 ;
fs/proc/loadavg.c +0 -3
@@ -10,9 +10,6 @@#include <linux/seqlock.h> #include <linux/time.h> -#define LOAD_INT (x) ((x) >> FSHIFT) -#define LOAD_FRAC (x) LOAD_INT(((x) & (FIXED_1-1)) * 100) -static int loadavg_proc_show (struct seq_file *m,void *v) {unsigned long avnrun[3 ];
include/linux/sched/loadavg.h +17 -4
@@ -22,10 +22,23 @@ extern void get_avenrun(unsigned long *loads, unsigned long offset, int shift);#define EXP_5 2014/* 1/exp(5sec/5min) */ #define EXP_15 2037/* 1/exp(5sec/15min) */ -#define CALC_LOAD (load,exp,n) \ - load *= exp; \ - load += n*(FIXED_1-exp); \ - load >>= FSHIFT; +/* +* a1 = a0 * e + a * (1 - e) +*/ +static inline unsigned long +calc_load (unsigned long load,unsigned long exp,unsigned long active) +{ +unsigned long newload; + + newload = load * exp + active * (FIXED_1 - exp); +if (active >= load) + newload +=FIXED_1 -1 ; + +return newload /FIXED_1 ; +} + +#define LOAD_INT (x) ((x) >> FSHIFT) +#define LOAD_FRAC (x) LOAD_INT(((x) & (FIXED_1-1)) * 100)extern void calc_global_load (unsigned long ticks);
kernel/debug/kdb/kdb_main.c +1 -6
@@ -2556,16 +2556,11 @@ static int kdb_summary(int argc, const char **argv) }kdb_printf ("%02ld:%02ld\n" , val.uptime/(60 *60 ), (val.uptime/60 )%60 ); -/* lifted from fs/proc/proc_misc.c::loadavg_read_proc() */ - -#define LOAD_INT (x) ((x) >> FSHIFT) -#define LOAD_FRAC (x) LOAD_INT(((x) & (FIXED_1-1)) * 100)kdb_printf ("load avg %ld.%02ld %ld.%02ld %ld.%02ld\n" ,LOAD_INT (val.loads[0 ]),LOAD_FRAC (val.loads[0 ]),LOAD_INT (val.loads[1 ]),LOAD_FRAC (val.loads[1 ]),LOAD_INT (val.loads[2 ]),LOAD_FRAC (val.loads[2 ])); -#undef LOAD_INT -#undef LOAD_FRAC +/* Display in kilobytes */ #define K (x) ((x) << (PAGE_SHIFT - 10)) kdb_printf("\nMemTotal: %8lu kB\nMemFree: %8lu kB\n"
kernel/sched/loadavg.c +0 -15
@@ -91,21 +91,6 @@ long calc_load_fold_active(struct rq *this_rq, long adjust)return delta; } -/* -* a1 = a0 * e + a * (1 - e) -*/ -static unsigned long -calc_load (unsigned long load,unsigned long exp,unsigned long active) -{ -unsigned long newload; - - newload = load * exp + active * (FIXED_1 - exp); -if (active >= load) - newload +=FIXED_1 -1 ; - -return newload /FIXED_1 ; -} -#ifdef CONFIG_NO_HZ_COMMON /* * Handle NO_HZ for the global load-average.
Signature · 1 line
2.18.0
Andrew Morton
maintainerRe: [PATCH 4/9] sched: loadavg: consolidate LOAD_INT, LOAD_FRAC, CALC_LOAD
Quoted 2 lines · On Tue, 28 Aug 2018 13:22:53 -0400 Johannes Weiner <[email protected]> wrote:
There are several definitions of those functions/macros in places that mess with fixed-point load averages. Provide an official version.
missed blk-iolatency.c for some reason?
block/blk-iolatency.c renamed from block/blk-iolatency.c~sched-loadavg-consolidate-load_int-load_frac-calc_load-fix +3 -3
@@ -139,7 +139,7 @@ struct iolatency_grp {#define BLKIOLATENCY_MAX_WIN_SIZE NSEC_PER_SEC/* * These are the constants used to fake the fixed-point moving average -* calculation just like load average. The call to CALC_LOAD folds +* calculation just like load average. The call to calc_load() folds * (FIXED_1 (2048) - exp_factor) * new_sample into lat_avg. The sampling * window size is bucketed to try to approximately calculate average * latency such that 1/exp (decay rate) is [1 min, 2.5 min) when windows
@@ -503,7 +503,7 @@ static void iolatency_check_latencies(st lat_info = &parent->child_lat;/* -* CALC_LOAD takes in a number stored in fixed point representation. +* calc_load() takes in a number stored in fixed point representation. * Because we are using this for IO time in ns, the values stored * are significantly larger than the FIXED_1 denominator (2048). * Therefore, rounding errors in the calculation are negligible and
@@ -512,7 +512,7 @@ static void iolatency_check_latencies(st exp_idx =min_t (int,BLKIOLATENCY_NR_EXP_FACTORS -1 ,div64_u64 (iolat->cur_win_nsec,BLKIOLATENCY_EXP_BUCKET_SIZE )); -CALC_LOAD (iolat->lat_avg, iolatency_exp_factors[exp_idx], stat.mean); +calc_load (iolat->lat_avg, iolatency_exp_factors[exp_idx], stat.mean);/* Everything is ok and we don't need to adjust the scale. */ if (stat.mean <= iolat->min_lat_nsec &&
_
Johannes Weiner
maintainerRe: [PATCH 4/9] sched: loadavg: consolidate LOAD_INT, LOAD_FRAC, CALC_LOAD
Quoted 6 lines · Andrew Morton
Quoted 2 lines · On Tue, 28 Aug 2018 13:22:53 -0400 Johannes Weiner <[email protected]> wrote:
There are several definitions of those functions/macros in places that mess with fixed-point load averages. Provide an official version.
missed
blk-iolatency.cfor some reason?
Ah, that callsite came in with this merge window. Thanks for the fixup.
block/blk-iolatency.c static void iolatency_check_latencies(st · line 515
Quoted 8 lines
block/blk-iolatency.c renamed from block/blk-iolatency.c~sched-loadavg-consolidate-load_int-load_frac-calc_load-fix +1
-1@@ -512,7 +512,7 @@ static void iolatency_check_latencies(st exp_idx =min_t (int,BLKIOLATENCY_NR_EXP_FACTORS -1 ,div64_u64 (iolat->cur_win_nsec,BLKIOLATENCY_EXP_BUCKET_SIZE )); -CALC_LOAD (iolat->lat_avg, iolatency_exp_factors[exp_idx], stat.mean); +calc_load (iolat->lat_avg, iolatency_exp_factors[exp_idx], stat.mean);
The macro used to modify the avg parameter in place, but with the function we need an explicit assignment to update the variable:
---
block/blk-iolatency.c +2 -1
@@ -512,7 +512,8 @@ static void iolatency_check_latencies(struct iolatency_grp *iolat, u64 now) exp_idx =min_t (int,BLKIOLATENCY_NR_EXP_FACTORS -1 ,div64_u64 (iolat->cur_win_nsec,BLKIOLATENCY_EXP_BUCKET_SIZE )); -calc_load (iolat->lat_avg, iolatency_exp_factors[exp_idx], stat.mean); + iolat->lat_avg =calc_load (iolat->lat_avg, + iolatency_exp_factors[exp_idx], stat.mean);/* Everything is ok and we don't need to adjust the scale. */ if (stat.mean <= iolat->min_lat_nsec &&
Johannes Weiner
maintainerPATCH 5/9 sched: loadavg: make calc_load_n() public
In reply to Johannes WeinerIt's going to be used in a later patch. Keep the churn separate.
--- include/linux/sched/loadavg.h | 3 + kernel/sched/loadavg.c | 138 +++++++++++++++++----------------- 2 files changed, 72 insertions(+), 69 deletions(-)
include/linux/sched/loadavg.h +3 -0
@@ -37,6 +37,9 @@ calc_load(unsigned long load, unsigned long exp, unsigned long active)return newload /FIXED_1 ; } +extern unsigned long calc_load_n (unsigned long load,unsigned long exp, +unsigned long active,unsigned int n); +#define LOAD_INT (x) ((x) >> FSHIFT)#define LOAD_FRAC (x) LOAD_INT(((x) & (FIXED_1-1)) * 100)
kernel/sched/loadavg.c +69 -69
150 lines not on this page · raw message
Signature · 1 line
2.18.0
Johannes Weiner
maintainerPATCH 6/9 sched: sched.h: make rq locking and clock functions available in stats.h
In reply to Johannes Weinerkernel/sched/sched.h includes "stats.h" half-way through the file. The next patch introduces users of sched.h's rq locking functions and update_rq_clock() in kernel/sched/stats.h. Move those definitions up in the file so they are available in stats.h.
--- kernel/sched/sched.h | 164 +++++++++++++++++++++---------------------- 1 file changed, 82 insertions(+), 82 deletions(-)
kernel/sched/sched.h +82 -82
@@ -926,6 +926,8 @@ DECLARE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);#define cpu_curr (cpu) (cpu_rq(cpu)->curr)#define raw_rq () raw_cpu_ptr(&runqueues) +extern void update_rq_clock (struct rq *rq); +static inline u64 __rq_clock_broken (struct rq *rq) {return READ_ONCE (rq->clock);
@@ -1044,6 +1046,86 @@ static inline void rq_repin_lock(struct rq *rq, struct rq_flags *rf)#endif } +struct rq *__task_rq_lock (struct task_struct *p,struct rq_flags *rf) + __acquires(rq->lock); + +struct rq *task_rq_lock (struct task_struct *p,struct rq_flags *rf) + __acquires(p->pi_lock) + __acquires(rq->lock); + +static inline void __task_rq_unlock (struct rq *rq,struct rq_flags *rf) + __releases(rq->lock) +{ +rq_unpin_lock (rq, rf); +raw_spin_unlock (&rq->lock); +} + +static inline void +task_rq_unlock (struct rq *rq,struct task_struct *p,struct rq_flags *rf) + __releases(rq->lock) + __releases(p->pi_lock) +{ +rq_unpin_lock (rq, rf); +raw_spin_unlock (&rq->lock); +raw_spin_unlock_irqrestore (&p->pi_lock, rf->flags); +} + +static inline void +rq_lock_irqsave (struct rq *rq,struct rq_flags *rf) + __acquires(rq->lock) +{ +raw_spin_lock_irqsave (&rq->lock, rf->flags); +rq_pin_lock (rq, rf); +} + +static inline void +rq_lock_irq (struct rq *rq,struct rq_flags *rf) + __acquires(rq->lock) +{ +raw_spin_lock_irq (&rq->lock); +rq_pin_lock (rq, rf); +} + +static inline void +rq_lock (struct rq *rq,struct rq_flags *rf) + __acquires(rq->lock) +{ +raw_spin_lock (&rq->lock); +rq_pin_lock (rq, rf); +} + +static inline void +rq_relock (struct rq *rq,struct rq_flags *rf) + __acquires(rq->lock) +{ +raw_spin_lock (&rq->lock); +rq_repin_lock (rq, rf); +} + +static inline void +rq_unlock_irqrestore (struct rq *rq,struct rq_flags *rf) + __releases(rq->lock) +{ +rq_unpin_lock (rq, rf); +raw_spin_unlock_irqrestore (&rq->lock, rf->flags); +} + +static inline void +rq_unlock_irq (struct rq *rq,struct rq_flags *rf) + __releases(rq->lock) +{ +rq_unpin_lock (rq, rf); +raw_spin_unlock_irq (&rq->lock); +} + +static inline void +rq_unlock (struct rq *rq,struct rq_flags *rf) + __releases(rq->lock) +{ +rq_unpin_lock (rq, rf); +raw_spin_unlock (&rq->lock); +} +#ifdef CONFIG_NUMA enum numa_topology_type { NUMA_DIRECT,
@@ -1683,8 +1765,6 @@ static inline void sub_nr_running(struct rq *rq, unsigned count)sched_update_tick_dependency (rq); } -extern void update_rq_clock (struct rq *rq); -extern void activate_task (struct rq *rq,struct task_struct *p,int flags);extern void deactivate_task (struct rq *rq,struct task_struct *p,int flags);
@@ -1765,86 +1845,6 @@ static inline void sched_rt_avg_update(struct rq *rq, u64 rt_delta) { }static inline void sched_avg_update (struct rq *rq) { }#endif -struct rq *__task_rq_lock (struct task_struct *p,struct rq_flags *rf) - __acquires(rq->lock); - -struct rq *task_rq_lock (struct task_struct *p,struct rq_flags *rf) - __acquires(p->pi_lock) - __acquires(rq->lock); - -static inline void __task_rq_unlock (struct rq *rq,struct rq_flags *rf) - __releases(rq->lock) -{ -rq_unpin_lock (rq, rf); -raw_spin_unlock (&rq->lock); -} - -static inline void -task_rq_unlock (struct rq *rq,struct task_struct *p,struct rq_flags *rf) - __releases(rq->lock) - __releases(p->pi_lock) -{ -rq_unpin_lock (rq, rf); -raw_spin_unlock (&rq->lock); -raw_spin_unlock_irqrestore (&p->pi_lock, rf->flags); -} - -static inline void -rq_lock_irqsave (struct rq *rq,struct rq_flags *rf) - __acquires(rq->lock) -{ -raw_spin_lock_irqsave (&rq->lock, rf->flags); -rq_pin_lock (rq, rf); -} - -static inline void -rq_lock_irq (struct rq *rq,struct rq_flags *rf) - __acquires(rq->lock) -{ -raw_spin_lock_irq (&rq->lock); -rq_pin_lock (rq, rf); -} - -static inline void -rq_lock (struct rq *rq,struct rq_flags *rf) - __acquires(rq->lock) -{ -raw_spin_lock (&rq->lock); -rq_pin_lock (rq, rf); -} - -static inline void -rq_relock (struct rq *rq,struct rq_flags *rf) - __acquires(rq->lock) -{ -raw_spin_lock (&rq->lock); -rq_repin_lock (rq, rf); -} - -static inline void -rq_unlock_irqrestore (struct rq *rq,struct rq_flags *rf) - __releases(rq->lock) -{ -rq_unpin_lock (rq, rf); -raw_spin_unlock_irqrestore (&rq->lock, rf->flags); -} - -static inline void -rq_unlock_irq (struct rq *rq,struct rq_flags *rf) - __releases(rq->lock) -{ -rq_unpin_lock (rq, rf); -raw_spin_unlock_irq (&rq->lock); -} - -static inline void -rq_unlock (struct rq *rq,struct rq_flags *rf) - __releases(rq->lock) -{ -rq_unpin_lock (rq, rf); -raw_spin_unlock (&rq->lock); -} -#ifdef CONFIG_SMP #ifdef CONFIG_PREEMPT
Signature · 1 line
2.18.0
Johannes Weiner
maintainerPATCH 7/9 sched: introduce this_rq_lock_irq()
In reply to Johannes Weinerdo_sched_yield() disables IRQs, looks up this_rq() and locks it. The next patch is adding another site with the same pattern, so provide a convenience function for it.
--- kernel/sched/core.c | 4 +--- kernel/sched/sched.h | 12 ++++++++++++ 2 files changed, 13 insertions(+), 3 deletions(-)
kernel/sched/core.c +1 -3
@@ -4960,9 +4960,7 @@ static void do_sched_yield(void)struct rq_flags rf;struct rq *rq; -local_irq_disable (); - rq =this_rq (); -rq_lock (rq, &rf); + rq =this_rq_lock_irq (&rf);schedstat_inc (rq->yld_count); current->sched_class->yield_task (rq);
kernel/sched/sched.h +12 -0
@@ -1126,6 +1126,18 @@ rq_unlock(struct rq *rq, struct rq_flags *rf)raw_spin_unlock (&rq->lock); } +static inline struct rq * +this_rq_lock_irq (struct rq_flags *rf) + __acquires(rq->lock) +{ +struct rq *rq; + +local_irq_disable (); + rq =this_rq (); +rq_lock (rq, rf); +return rq; +} +#ifdef CONFIG_NUMA enum numa_topology_type { NUMA_DIRECT,
Signature · 1 line
2.18.0
Johannes Weiner
maintainerPATCH 8/9 psi: pressure stall information for CPU, memory, and IO
In reply to Johannes WeinerWhen systems are overcommitted and resources become contended, it's hard to tell exactly the impact this has on workload productivity, or how close the system is to lockups and OOM kills. In particular, when machines work multiple jobs concurrently, the impact of overcommit in terms of latency and throughput on the individual job can be enormous.
In order to maximize hardware utilization without sacrificing individual job health or risk complete machine lockups, this patch implements a way to quantify resource pressure in the system.
A kernel built with CONFIG_PSI=y creates files in /proc/pressure/ that expose the percentage of time the system is stalled on CPU, memory, or IO, respectively. Stall states are aggregate versions of the per-task delay accounting delays:
cpu: some tasks are runnable but not executing on a CPU
memory: tasks are reclaiming, or waiting for swapin or thrashing cache
io: tasks are waiting for io completions
These percentages of walltime can be thought of as pressure percentages, and they give a general sense of system health and productivity loss incurred by resource overcommit. They can also indicate when the system is approaching lockup scenarios and OOMs.
To do this, psi keeps track of the task states associated with each CPU and samples the time they spend in stall states. Every 2 seconds, the samples are averaged across CPUs - weighted by the CPUs' non-idle time to eliminate artifacts from unused CPUs - and translated into percentages of walltime. A running average of those percentages is maintained over 10s, 1m, and 5m periods (similar to the loadaverage).
v2:
- stable clock tick, as per Peter
- data structure layout optimization, as per Peter
- fix u64 divisions on 32 bit, as per Peter
- outermost psi_disabled checks, as per Peter
- coding style fixes, as per Peter
- just-in-time stats aggregation, as per Suren
- fix task state corruption with CONFIG_PREEMPT, as per Suren
- CONFIG_PSI=n build error
- avoid writing p->sched_psi_wake_requeue unnecessarily
- documentation & comment updates
v3:
- pack scheduler hotpath data into one cacheline, as per Peter and Linus
- drop unnecessary SCHED_INFO dependency, as per Peter
- lockless live-state aggregation, as per Peter
- do_div -> div64_ul and some other cleanups, as per Peter
- realtime sampling period and slipped sample handling, as per Tejun
v4:
- replace an unsafe cpu_curr() dereference in the aggregator by sampling active reclaimers from scheduler_tick(), as per Peter
- fix several race conditions that cause the unlocked live aggregator to get ahead of the scheduler's recorded times and cause sample calculations to underflow into bogusly large time deltas, as per Suren
- fix rare accounting artifacts from CPU hotplugging, as per Peter
- make the aggregation loop over all states more readable, as per Peter
--- Documentation/accounting/psi.txt | 64 +++ include/linux/psi.h | 28 ++ include/linux/psi_types.h | 92 +++++ include/linux/sched.h | 10 + init/Kconfig | 15 + kernel/fork.c | 4 + kernel/sched/Makefile | 1 + kernel/sched/core.c | 12 +- kernel/sched/psi.c | 650 +++++++++++++++++++++++++++++++ kernel/sched/sched.h | 2 + kernel/sched/stats.h | 86 ++++ mm/compaction.c | 5 + mm/filemap.c | 15 +- mm/page_alloc.c | 9 + mm/vmscan.c | 9 + 15 files changed, 996 insertions(+), 6 deletions(-) create mode 100644 Documentation/accounting/psi.txt create mode 100644 include/linux/psi.h create mode 100644 include/linux/psi_types.h create mode 100644 kernel/sched/psi.c
Documentation/accounting/psi.txt new file +64 -0
@@ -0,0 +1,64 @@ +================================ +PSI - Pressure Stall Information +================================ + +:Date: April, 2018 +:Author: Johannes Weiner <[email protected]> + +When CPU, memory or IO devices are contended, workloads experience +latency spikes, throughput losses, and run the risk of OOM kills. + +Without an accurate measure of such contention, users are forced to +either play it safe and under-utilize their hardware resources, or +roll the dice and frequently suffer the disruptions resulting from +excessive overcommit. + +The psi feature identifies and quantifies the disruptions caused by +such resource crunches and the time impact it has on complex workloads +or even entire systems. + +Having an accurate measure of productivity losses caused by resource +scarcity aids users in sizing workloads to hardware--or provisioning +hardware according to workload demand. + +As psi aggregates this information in realtime, systems can be managed +dynamically using techniques such as load shedding, migrating jobs to +other systems or data centers, or strategically pausing or killing low +priority or restartable batch jobs. + +This allows maximizing hardware utilization without sacrificing +workload health or risking major disruptions such as OOM kills. + +Pressure interface +================== + +Pressure information for each resource is exported through the +respective file in /proc/pressure/ -- cpu, memory, and io. + +In both cases, the format for CPU is as such: + +some avg10=0.00 avg60=0.00 avg300=0.00 total=0 + +and for memory and IO: + +some avg10=0.00 avg60=0.00 avg300=0.00 total=0 +full avg10=0.00 avg60=0.00 avg300=0.00 total=0 + +The "some" line indicates the share of time in which at least some +tasks are stalled on a given resource. + +The "full" line indicates the share of time in which all non-idle +tasks are stalled on a given resource simultaneously. In this state +actual CPU cycles are going to waste, and a workload that spends +extended time in this state is considered to be thrashing. This has +severe impact on performance, and it's useful to distinguish this +situation from a state where some tasks are stalled but the CPU is +still doing productive work. As such, time spent in this subset of the +stall state is tracked separately and exported in the "full" averages. + +The ratios are tracked as recent trends over ten, sixty, and three +hundred second windows, which gives insight into short term events as +well as medium and long term trends. The total absolute stall time is +tracked and exported as well, to allow detection of latency spikes +which wouldn't necessarily make a dent in the time averages, or to +average trends over custom time frames.
include/linux/psi.h new file +28 -0
@@ -0,0 +1,28 @@ +#ifndef _LINUX_PSI_H +#define _LINUX_PSI_H + +#include <linux/psi_types.h> +#include <linux/sched.h> + +#ifdef CONFIG_PSI + +extern bool psi_disabled; + +void psi_init (void ); + +void psi_task_change (struct task_struct *task,int clear,int set); + +void psi_memstall_tick (struct task_struct *task,int cpu); +void psi_memstall_enter (unsigned long *flags); +void psi_memstall_leave (unsigned long *flags); + +#else /* CONFIG_PSI */ + +static inline void psi_init (void ) {} + +static inline void psi_memstall_enter (unsigned long *flags) {} +static inline void psi_memstall_leave (unsigned long *flags) {} + +#endif /* CONFIG_PSI */ + +#endif /* _LINUX_PSI_H */
include/linux/psi_types.h new file +92 -0
92 lines not on this page · raw message
include/linux/sched.h +10 -0
@@ -25,6 +25,7 @@#include <linux/latencytop.h> #include <linux/sched/prio.h> #include <linux/signal_types.h> +#include <linux/psi_types.h> #include <linux/mm_types_task.h> #include <linux/task_io_accounting.h> #include <linux/rseq.h>
@@ -710,6 +711,10 @@ struct task_struct { unsigned sched_contributes_to_load:1; unsigned sched_migrated:1; unsigned sched_remote_wakeup:1; +#ifdef CONFIG_PSI + unsigned sched_psi_wake_requeue:1; +#endif +/* Force alignment to the next boundary: */ unsigned :0;
@@ -957,6 +962,10 @@ struct task_struct {siginfo_t *last_siginfo;struct task_io_accounting ioac; +#ifdef CONFIG_PSI +/* Pressure stall state */ +unsigned int psi_flags; +#endif #ifdef CONFIG_TASK_XACCT /* Accumulated RSS usage: */ u64 acct_rss_mem1;
@@ -1397,6 +1406,7 @@ extern struct pid *cad_pid;#define PF_KTHREAD 0x00200000/* I am a kernel thread */ #define PF_RANDOMIZE 0x00400000/* Randomize virtual address space */ #define PF_SWAPWRITE 0x00800000/* Allowed to write to swap */ +#define PF_MEMSTALL 0x01000000/* Stalled due to lack of memory */ #define PF_NO_SETAFFINITY 0x04000000/* Userland is not allowed to meddle with cpus_allowed */ #define PF_MCE_EARLY 0x08000000/* Early kill for mce process policy */ #define PF_MUTEX_TESTER 0x20000000/* Thread belongs to the rt mutex tester */
init/Kconfig +15 -0
@@ -455,6 +455,21 @@ config TASK_IO_ACCOUNTING Say N if unsure. +config PSI + bool "Pressure stall information tracking" + help + Collect metrics that indicate how overcommitted the CPU, memory, + and IO capacity are in the system. + + If you say Y here, the kernel will create /proc/pressure/ with the + pressure statistics files cpu, memory, and io. These will indicate + the share of walltime in which some or all tasks in the system are + delayed due to contention of the respective resource. + + For more details see Documentation/accounting/psi.txt. + + Say N if unsure. + endmenu # "CPU/Task time and stats accounting" config CPU_ISOLATION
kernel/fork.c +4 -0
@@ -1736,6 +1736,10 @@ static __latent_entropy struct task_struct *copy_process( p->default_timer_slack_ns = current->timer_slack_ns; +#ifdef CONFIG_PSI + p->psi_flags =0 ; +#endif +task_io_accounting_init (&p->ioac);acct_clear_integrals (p);
kernel/sched/Makefile +1 -0
@@ -29,3 +29,4 @@ obj-$(CONFIG_CPU_FREQ) += cpufreq.o obj-$(CONFIG_CPU_FREQ_GOV_SCHEDUTIL) += cpufreq_schedutil.o obj-$(CONFIG_MEMBARRIER) += membarrier.o obj-$(CONFIG_CPU_ISOLATION) += isolation.o +obj-$(CONFIG_PSI) += psi.o
kernel/sched/core.c +10 -2
@@ -744,8 +744,10 @@ static inline void enqueue_task(struct rq *rq, struct task_struct *p, int flags)if (!(flags &ENQUEUE_NOCLOCK ))update_rq_clock (rq); -if (!(flags &ENQUEUE_RESTORE )) +if (!(flags &ENQUEUE_RESTORE )) {sched_info_queued (rq, p); +psi_enqueue (p, flags &ENQUEUE_WAKEUP ); + } p->sched_class->enqueue_task (rq, p, flags); }
@@ -755,8 +757,10 @@ static inline void dequeue_task(struct rq *rq, struct task_struct *p, int flags)if (!(flags &DEQUEUE_NOCLOCK ))update_rq_clock (rq); -if (!(flags &DEQUEUE_SAVE )) +if (!(flags &DEQUEUE_SAVE )) {sched_info_dequeued (rq, p); +psi_dequeue (p, flags &DEQUEUE_SLEEP ); + } p->sched_class->dequeue_task (rq, p, flags); }
@@ -2060,6 +2064,7 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) cpu =select_task_rq (p, p->wake_cpu,SD_BALANCE_WAKE , wake_flags);if (task_cpu (p) != cpu) { wake_flags |=WF_MIGRATED ; +psi_ttwu_dequeue (p);set_task_cpu (p, cpu); }
@@ -3078,6 +3083,7 @@ void scheduler_tick(void) curr->sched_class->task_tick (rq, curr,0 );cpu_load_update_active (rq);calc_global_load_tick (rq); +psi_task_tick (rq);rq_unlock (rq, &rf);
@@ -6110,6 +6116,8 @@ void __init sched_init(void)init_schedstats (); +psi_init (); + scheduler_running =1 ; }
kernel/sched/psi.c new file +650 -0
650 lines not on this page · raw message
kernel/sched/sched.h +2 -0
@@ -54,6 +54,7 @@#include <linux/proc_fs.h> #include <linux/prefetch.h> #include <linux/profile.h> +#include <linux/psi.h> #include <linux/rcupdate_wait.h> #include <linux/security.h> #include <linux/stackprotector.h>
@@ -320,6 +321,7 @@ extern bool dl_cpu_busy(unsigned int cpu);#ifdef CONFIG_CGROUP_SCHED #include <linux/cgroup.h> +#include <linux/psi.h> struct cfs_rq ;struct rt_rq ;
kernel/sched/stats.h +86 -0
92 lines not on this page · raw message
mm/compaction.c +5 -0
@@ -22,6 +22,7 @@#include <linux/kthread.h> #include <linux/freezer.h> #include <linux/page_owner.h> +#include <linux/psi.h> #include "internal.h" #ifdef CONFIG_COMPACTION
@@ -2068,11 +2069,15 @@ static int kcompactd(void *p) pgdat->kcompactd_classzone_idx = pgdat->nr_zones -1 ;while (!kthread_should_stop ()) { +unsigned long pflags; +trace_mm_compaction_kcompactd_sleep (pgdat->node_id);wait_event_freezable (pgdat->kcompactd_wait,kcompactd_work_requested (pgdat)); +psi_memstall_enter (&pflags);kcompactd_do_work (pgdat); +psi_memstall_leave (&pflags); }return 0 ;
mm/filemap.c +11 -4
@@ -37,6 +37,7 @@#include <linux/shmem_fs.h> #include <linux/rmap.h> #include <linux/delayacct.h> +#include <linux/psi.h> #include "internal.h" #define CREATE_TRACE_POINTS
@@ -1075,11 +1076,14 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q,struct wait_page_queue wait_page;wait_queue_entry_t *wait = &wait_page.wait;bool thrashing = false; +unsigned long pflags;int ret =0 ; -if (bit_nr == PG_locked && !PageSwapBacked (page) && +if (bit_nr == PG_locked && !PageUptodate (page) &&PageWorkingset (page)) { -delayacct_thrashing_start (); +if (!PageSwapBacked (page)) +delayacct_thrashing_start (); +psi_memstall_enter (&pflags); thrashing = true; }
@@ -1121,8 +1125,11 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q,finish_wait (q, wait); -if (thrashing) -delayacct_thrashing_end (); +if (thrashing) { +if (!PageSwapBacked (page)) +delayacct_thrashing_end (); +psi_memstall_leave (&pflags); + }/* * A signal could leave PageWaiters set. Clearing it here if
mm/page_alloc.c +9 -0
@@ -67,6 +67,7 @@#include <linux/ftrace.h> #include <linux/lockdep.h> #include <linux/nmi.h> +#include <linux/psi.h> #include <asm/sections.h> #include <asm/tlbflush.h>
@@ -3549,15 +3550,20 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, enum compact_priority prio, enum compact_result *compact_result) { struct page *page; +unsigned long pflags;unsigned int noreclaim_flag;if (!order)return NULL ; +psi_memstall_enter (&pflags); noreclaim_flag =memalloc_noreclaim_save (); + *compact_result =try_to_compact_pages (gfp_mask, order, alloc_flags, ac, prio); +memalloc_noreclaim_restore (noreclaim_flag); +psi_memstall_leave (&pflags);if (*compact_result <=COMPACT_INACTIVE )return NULL ;
@@ -3756,11 +3762,13 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order,struct reclaim_state reclaim_state;int progress;unsigned int noreclaim_flag; +unsigned long pflags;cond_resched ();/* We now go into synchronous reclaim */ cpuset_memory_pressure_bump (); +psi_memstall_enter (&pflags);fs_reclaim_acquire (gfp_mask); noreclaim_flag =memalloc_noreclaim_save (); reclaim_state.reclaimed_slab =0 ;
@@ -3772,6 +3780,7 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order, current->reclaim_state =NULL ;memalloc_noreclaim_restore (noreclaim_flag);fs_reclaim_release (gfp_mask); +psi_memstall_leave (&pflags);cond_resched ();
mm/vmscan.c +9 -0
@@ -49,6 +49,7 @@#include <linux/prefetch.h> #include <linux/printk.h> #include <linux/dax.h> +#include <linux/psi.h> #include <asm/tlbflush.h> #include <asm/div64.h>
@@ -3131,6 +3132,7 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, {struct zonelist *zonelist;unsigned long nr_reclaimed; +unsigned long pflags;int nid;unsigned int noreclaim_flag;struct scan_control sc = {
@@ -3159,9 +3161,13 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, sc.gfp_mask, sc.reclaim_idx); +psi_memstall_enter (&pflags); noreclaim_flag =memalloc_noreclaim_save (); + nr_reclaimed =do_try_to_free_pages (zonelist, &sc); +memalloc_noreclaim_restore (noreclaim_flag); +psi_memstall_leave (&pflags);trace_mm_vmscan_memcg_reclaim_end (nr_reclaimed);
@@ -3326,6 +3332,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx)int i;unsigned long nr_soft_reclaimed;unsigned long nr_soft_scanned; +unsigned long pflags;struct zone *zone;struct scan_control sc = { .gfp_mask =GFP_KERNEL ,
@@ -3336,6 +3343,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx) .may_swap = 1, }; +psi_memstall_enter (&pflags);__fs_reclaim_acquire ();count_vm_event (PAGEOUTRUN );
@@ -3437,6 +3445,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx) out:snapshot_refaults (NULL , pgdat);__fs_reclaim_release (); +psi_memstall_leave (&pflags);/* * Return the order kswapd stopped reclaiming at as * prepare_kswapd_sleep() takes it into account. If another caller
Signature · 1 line
2.18.0
Randy Dunlap
reviewerRe: [PATCH 8/9] psi: pressure stall information for CPU, memory, and IO
Quoted 33 lines · Johannes Weiner
Documentation/accounting/psi.txt new file +27
-0@@ -0,0 +1,64 @@ +================================ +PSI - Pressure Stall Information +================================ + +:Date: April, 2018 +:Author: Johannes Weiner <[email protected]> + +When CPU, memory or IO devices are contended, workloads experience +latency spikes, throughput losses, and run the risk of OOM kills. + +Without an accurate measure of such contention, users are forced to +either play it safe and under-utilize their hardware resources, or +roll the dice and frequently suffer the disruptions resulting from +excessive overcommit. + +The psi feature identifies and quantifies the disruptions caused by +such resource crunches and the time impact it has on complex workloads +or even entire systems. + +Having an accurate measure of productivity losses caused by resource +scarcity aids users in sizing workloads to hardware--or provisioning +hardware according to workload demand. + +As psi aggregates this information in realtime, systems can be managed +dynamically using techniques such as load shedding, migrating jobs to +other systems or data centers, or strategically pausing or killing low +priority or restartable batch jobs.
Documentation/accounting/psi.txt line 37
Quoted 10 lines
Documentation/accounting/psi.txt new file +10
-0+ +This allows maximizing hardware utilization without sacrificing +workload health or risking major disruptions such as OOM kills. + +Pressure interface +================== + +Pressure information for each resource is exported through the +respective file in /proc/pressure/ -- cpu, memory, and io. +
Hi,
Documentation/accounting/psi.txt line 38
Quoted 1 line
Documentation/accounting/psi.txt new file +1
-0+In both cases, the format for CPU is as such:
I don't see what "In both cases" refers to here. It seems that you could just remove it.
Quoted 10 lines
Documentation/accounting/psi.txt new file +10
-0+ +some avg10=0.00 avg60=0.00 avg300=0.00 total=0 + +and for memory and IO: + +some avg10=0.00 avg60=0.00 avg300=0.00 total=0 +full avg10=0.00 avg60=0.00 avg300=0.00 total=0 + +The "some" line indicates the share of time in which at least some +tasks are stalled on a given resource.
Signature · 1 line
~Randy
Johannes Weiner
maintainerRe: [PATCH 8/9] psi: pressure stall information for CPU, memory, and IO
Quoted 47 lines · Randy Dunlap
Quoted 33 lines · Johannes Weiner
Documentation/accounting/psi.txt new file +27
-0@@ -0,0 +1,64 @@ +================================ +PSI - Pressure Stall Information +================================ + +:Date: April, 2018 +:Author: Johannes Weiner <[email protected]> + +When CPU, memory or IO devices are contended, workloads experience +latency spikes, throughput losses, and run the risk of OOM kills. + +Without an accurate measure of such contention, users are forced to +either play it safe and under-utilize their hardware resources, or +roll the dice and frequently suffer the disruptions resulting from +excessive overcommit. + +The psi feature identifies and quantifies the disruptions caused by +such resource crunches and the time impact it has on complex workloads +or even entire systems. + +Having an accurate measure of productivity losses caused by resource +scarcity aids users in sizing workloads to hardware--or provisioning +hardware according to workload demand. + +As psi aggregates this information in realtime, systems can be managed +dynamically using techniques such as load shedding, migrating jobs to +other systems or data centers, or strategically pausing or killing low +priority or restartable batch jobs.Documentation/accounting/psi.txt line 37
Quoted 10 lines
Documentation/accounting/psi.txt new file +10
-0+ +This allows maximizing hardware utilization without sacrificing +workload health or risking major disruptions such as OOM kills. + +Pressure interface +================== + +Pressure information for each resource is exported through the +respective file in /proc/pressure/ -- cpu, memory, and io. +Hi,
Quoted 4 lines
Documentation/accounting/psi.txt line 38
Quoted 1 line
Documentation/accounting/psi.txt new file +1
-0+In both cases, the format for CPU is as such:I don't see what "In both cases" refers to here. It seems that you could just remove it.
You're right, that must be a left-over from when I described CPU separately; "both cases" referred to memory and IO which have identical formats. It needs to be removed:
Documentation/accounting/psi.txt +1 -1
@@ -35,7 +35,7 @@ Pressure interface Pressure information for each resource is exported through the respective file in /proc/pressure/ -- cpu, memory, and io. -In both cases, the format for CPU is as such: +The format for CPU is as such: some avg10=0.00 avg60=0.00 avg300=0.00 total=0
Randy Dunlap
reviewerRe: [PATCH 8/9] psi: pressure stall information for CPU, memory, and IO
Quoted 62 lines · Johannes Weiner
Quoted 47 lines · Randy Dunlap
Quoted 33 lines · Johannes Weiner
Documentation/accounting/psi.txt new file +27
-0@@ -0,0 +1,64 @@ +================================ +PSI - Pressure Stall Information +================================ + +:Date: April, 2018 +:Author: Johannes Weiner <[email protected]> + +When CPU, memory or IO devices are contended, workloads experience +latency spikes, throughput losses, and run the risk of OOM kills. + +Without an accurate measure of such contention, users are forced to +either play it safe and under-utilize their hardware resources, or +roll the dice and frequently suffer the disruptions resulting from +excessive overcommit. + +The psi feature identifies and quantifies the disruptions caused by +such resource crunches and the time impact it has on complex workloads +or even entire systems. + +Having an accurate measure of productivity losses caused by resource +scarcity aids users in sizing workloads to hardware--or provisioning +hardware according to workload demand. + +As psi aggregates this information in realtime, systems can be managed +dynamically using techniques such as load shedding, migrating jobs to +other systems or data centers, or strategically pausing or killing low +priority or restartable batch jobs.Documentation/accounting/psi.txt line 37
Quoted 10 lines
Documentation/accounting/psi.txt new file +10
-0+ +This allows maximizing hardware utilization without sacrificing +workload health or risking major disruptions such as OOM kills. + +Pressure interface +================== + +Pressure information for each resource is exported through the +respective file in /proc/pressure/ -- cpu, memory, and io. +Hi,
Quoted 4 lines
Documentation/accounting/psi.txt line 38
Quoted 1 line
Documentation/accounting/psi.txt new file +1
-0+In both cases, the format for CPU is as such:I don't see what "In both cases" refers to here. It seems that you could just remove it.
You're right, that must be a left-over from when I described CPU separately; "both cases" referred to memory and IO which have identical formats. It needs to be removed:
Documentation/accounting/psi.txt +0
-0
Documentation/accounting/psi.txt Pressure interface · line 40
Quoted 7 lines
Documentation/accounting/psi.txt +1
-1@@ -35,7 +35,7 @@ Pressure interface Pressure information for each resource is exported through the respective file in /proc/pressure/ -- cpu, memory, and io. -In both cases, the format for CPU is as such: +The format for CPU is as such: some avg10=0.00 avg60=0.00 avg300=0.00 total=0
OK. However, after reading patch 9/9, I thought that the "both cases" could possibly mean the files in /proc/pressure/ and the files in cgroup ({cpu,io,memory}.pressure).
Signature · 1 line
~Randy
Peter Zijlstra
maintainerRe: [PATCH 8/9] psi: pressure stall information for CPU, memory, and IO
In reply to Johannes WeinerQuoted 64 lines · Johannes Weiner
(quoted diff) +62
-0+enum psi_states { + PSI_IO_SOME, + PSI_IO_FULL, + PSI_MEM_SOME, + PSI_MEM_FULL, + PSI_CPU_SOME, + /* Only per-CPU, to weigh the CPU in the global average: */ + PSI_NONIDLE, + NR_PSI_STATES, +}; +static u32 get_recent_time(struct psi_group *group, int cpu, + enum psi_states state) +{ + struct psi_group_cpu *groupc = per_cpu_ptr(group->pcpu, cpu); + unsigned int seq; + u32 time, delta; + + do { + seq = read_seqcount_begin(&groupc->seq); + + time = groupc->times[state]; + /* + * In addition to already concluded states, we also + * incorporate currently active states on the CPU, + * since states may last for many sampling periods. + * + * This way we keep our delta sampling buckets small + * (u32) and our reported pressure close to what's + * actually happening. + */ + if (test_state(groupc->tasks, state)) + time += cpu_clock(cpu) - groupc->state_start; + } while (read_seqcount_retry(&groupc->seq, seq)); + + delta = time - groupc->times_prev[state]; + groupc->times_prev[state] = time; + + return delta; +} +static bool update_stats(struct psi_group *group) +{ + u64 deltas[NR_PSI_STATES - 1] = { 0, }; + unsigned long missed_periods = 0; + unsigned long nonidle_total = 0; + u64 now, expires, period; + int cpu; + int s; + + mutex_lock(&group->stat_lock); + + /* + * Collect the per-cpu time buckets and average them into a + * single time sample that is normalized to wallclock time. + * + * For averaging, each CPU is weighted by its non-idle time in + * the sampling period. This eliminates artifacts from uneven + * loading, or even entirely idle CPUs. + */ + for_each_possible_cpu(cpu) { + u32 nonidle; + + nonidle = get_recent_time(group, cpu, PSI_NONIDLE);
Review
Quoted 10 lines
(quoted diff) +10
-0+ nonidle = nsecs_to_jiffies(nonidle); + nonidle_total += nonidle; + + for (s = 0; s < PSI_NONIDLE; s++) { + u32 delta; + + delta = get_recent_time(group, cpu, s); + deltas[s] += (u64)delta * nonidle; + } + }
This does the whole seqcount thing 6x, which is a bit of a waste.
struct snapshot {
u32 times[NR_PSI_STATES];
};static inline struct snapshot get_times_snapshot(struct psi_group *pg, int cpu)
{
struct pci_group_cpu *pgc = per_cpu_ptr(pg->pcpu, cpu);
struct snapshot s;
unsigned int seq;
u32 delta;
int i;
do {
seq = read_seqcount_begin(&pgc->seq);
delta = cpu_clock(cpu) - pgc->state_start;
for (i = 0; i < NR_PSI_STATES; i++) {
s.times[i] = gpc->times[i];
if (test_state(pgc->tasks, i))
s.times[i] += delta;
}
} while (read_seqcount_retry(&pgc->seq, seq);
return s;
} for_each_possible_cpu(cpu) {
struct snapshot s = get_times_snapshot(pg, cpu);
nonidle = nsecs_to_jiffies(s.times[PSI_NONIDLE]);
nonidle_total += nonidle;
for (i = 0; i < PSI_NONIDLE; i++)
deltas[s] += (u64)s.times[i] * nonidle;
/* ... */
}It's a bit cumbersome, but that's because of C.
Peter Zijlstra
maintainerRe: [PATCH 8/9] psi: pressure stall information for CPU, memory, and IO
Quoted 17 lines · Peter Zijlstra
This does the whole seqcount thing 6x, which is a bit of a waste.
struct snapshot { u32 times[NR_PSI_STATES]; };static inline struct snapshot
get_times_snapshot(struct psi_group *pg, int cpu){ struct pci_group_cpu *pgc = per_cpu_ptr(pg->pcpu, cpu); struct snapshot s; unsigned int seq; u32 delta; int i; do { seq = read_seqcount_begin(&pgc->seq);
Quoted 8 lines
delta = cpu_clock(cpu) - pgc->state_start;
for (i = 0; i < NR_PSI_STATES; i++) {
s.times[i] = gpc->times[i];
if (test_state(pgc->tasks, i))
s.times[i] += delta;
}
} while (read_seqcount_retry(&pgc->seq, seq);Sorry, I forgot the whole times_prev thing:
for (i = 0; i < NR_PSI_STATES; i++) {
tmp = s.times[i];
s.times[i] -= pgc->times_prev[i];
pgc->times_prev[i] = tmp;
}Quoted 19 lines
return s; }for_each_possible_cpu(cpu) { struct snapshot s = get_times_snapshot(pg, cpu); nonidle = nsecs_to_jiffies(s.times[PSI_NONIDLE]); nonidle_total += nonidle; for (i = 0; i < PSI_NONIDLE; i++) deltas[s] += (u64)s.times[i] * nonidle; /* ... */ }It's a bit cumbersome, but that's because of C.
Johannes Weiner
maintainerRe: [PATCH 8/9] psi: pressure stall information for CPU, memory, and IO
In reply to Peter ZijlstraQuoted 77 lines · Peter Zijlstra
Quoted 64 lines · Johannes Weiner
(quoted diff) +62
-0+enum psi_states { + PSI_IO_SOME, + PSI_IO_FULL, + PSI_MEM_SOME, + PSI_MEM_FULL, + PSI_CPU_SOME, + /* Only per-CPU, to weigh the CPU in the global average: */ + PSI_NONIDLE, + NR_PSI_STATES, +}; +static u32 get_recent_time(struct psi_group *group, int cpu, + enum psi_states state) +{ + struct psi_group_cpu *groupc = per_cpu_ptr(group->pcpu, cpu); + unsigned int seq; + u32 time, delta; + + do { + seq = read_seqcount_begin(&groupc->seq); + + time = groupc->times[state]; + /* + * In addition to already concluded states, we also + * incorporate currently active states on the CPU, + * since states may last for many sampling periods. + * + * This way we keep our delta sampling buckets small + * (u32) and our reported pressure close to what's + * actually happening. + */ + if (test_state(groupc->tasks, state)) + time += cpu_clock(cpu) - groupc->state_start; + } while (read_seqcount_retry(&groupc->seq, seq)); + + delta = time - groupc->times_prev[state]; + groupc->times_prev[state] = time; + + return delta; +} +static bool update_stats(struct psi_group *group) +{ + u64 deltas[NR_PSI_STATES - 1] = { 0, }; + unsigned long missed_periods = 0; + unsigned long nonidle_total = 0; + u64 now, expires, period; + int cpu; + int s; + + mutex_lock(&group->stat_lock); + + /* + * Collect the per-cpu time buckets and average them into a + * single time sample that is normalized to wallclock time. + * + * For averaging, each CPU is weighted by its non-idle time in + * the sampling period. This eliminates artifacts from uneven + * loading, or even entirely idle CPUs. + */ + for_each_possible_cpu(cpu) { + u32 nonidle; + + nonidle = get_recent_time(group, cpu, PSI_NONIDLE);Review
Quoted 10 lines
(quoted diff) +10
-0+ nonidle = nsecs_to_jiffies(nonidle); + nonidle_total += nonidle; + + for (s = 0; s < PSI_NONIDLE; s++) { + u32 delta; + + delta = get_recent_time(group, cpu, s); + deltas[s] += (u64)delta * nonidle; + } + }This does the whole seqcount thing 6x, which is a bit of a waste.
[...]
Quoted 1 line
It's a bit cumbersome, but that's because of C.
I was actually debating exactly this with Suren before, but since this is a super cold path I went with readability. I was also thinking that restarts could happen quite regularly under heavy scheduler load, and so keeping the individual retry sections small could be helpful - but I didn't instrument this in any way.
No strong opinion from me, I can send an updated patch if you prefer.
Peter Zijlstra
maintainerRe: [PATCH 8/9] psi: pressure stall information for CPU, memory, and IO
Quoted 11 lines · Johannes Weiner
Quoted 1 line
This does the whole seqcount thing 6x, which is a bit of a waste.
[...]
Quoted 1 line
It's a bit cumbersome, but that's because of C.
I was actually debating exactly this with Suren before, but since this is a super cold path I went with readability. I was also thinking that restarts could happen quite regularly under heavy scheduler load, and so keeping the individual retry sections small could be helpful - but I didn't instrument this in any way.
I was hoping going over the whole thing once would reduce the time we need to keep that line in shared mode and reduce traffic. And yes, this path is cold, but I was thinking about reducing the interference on the remote CPU.
Alternatively, we memcpy the whole line under the seqlock and then do everything later.
Also, this only has a single cpu_clock() invocation.
Johannes Weiner
maintainerRe: [PATCH 8/9] psi: pressure stall information for CPU, memory, and IO
Quoted 19 lines · Peter Zijlstra
Quoted 11 lines · Johannes Weiner
Quoted 1 line
This does the whole seqcount thing 6x, which is a bit of a waste.
[...]
Quoted 1 line
It's a bit cumbersome, but that's because of C.
I was actually debating exactly this with Suren before, but since this is a super cold path I went with readability. I was also thinking that restarts could happen quite regularly under heavy scheduler load, and so keeping the individual retry sections small could be helpful - but I didn't instrument this in any way.
I was hoping going over the whole thing once would reduce the time we need to keep that line in shared mode and reduce traffic. And yes, this path is cold, but I was thinking about reducing the interference on the remote CPU.
Quoted 4 lines
Alternatively, we memcpy the whole line under the seqlock and then do everything later.
Also, this only has a single
cpu_clock()invocation.
Good points.
How about the below? It's still pretty readable, and generates compact code inside the now single retry section:
ffffffff81ed464f: 44 89 ff mov %r15d,%edi
ffffffff81ed4652: e8 00 00 00 00 callq ffffffff81ed4657 <update_stats+0xca>
ffffffff81ed4653: R_X86_64_PLT32 sched_clock_cpu-0x4
memcpy(times, groupc->times, sizeof(groupc->times));
ffffffff81ed4657: 49 8b 14 24 mov (%r12),%rdx
state_start = groupc->state_start;
ffffffff81ed465b: 48 8b 4b 50 mov 0x50(%rbx),%rcx
memcpy(times, groupc->times, sizeof(groupc->times));
ffffffff81ed465f: 48 89 54 24 30 mov %rdx,0x30(%rsp)
ffffffff81ed4664: 49 8b 54 24 08 mov 0x8(%r12),%rdx
ffffffff81ed4669: 48 89 54 24 38 mov %rdx,0x38(%rsp)
ffffffff81ed466e: 49 8b 54 24 10 mov 0x10(%r12),%rdx
ffffffff81ed4673: 48 89 54 24 40 mov %rdx,0x40(%rsp)
memcpy(tasks, groupc->tasks, sizeof(groupc->tasks));
ffffffff81ed4678: 49 8b 55 00 mov 0x0(%r13),%rdx
ffffffff81ed467c: 48 89 54 24 24 mov %rdx,0x24(%rsp)
ffffffff81ed4681: 41 8b 55 08 mov 0x8(%r13),%edx
ffffffff81ed4685: 89 54 24 2c mov %edx,0x2c(%rsp)---
kernel/sched/psi.c +25 -18
@@ -197,17 +197,26 @@ static bool test_state(unsigned int *tasks, enum psi_states state) } } -static u32 get_recent_time (struct psi_group *group,int cpu, -enum psi_states state) +static void get_recent_times (struct psi_group *group,int cpu,u32 *times) {struct psi_group_cpu *groupc =per_cpu_ptr (group->pcpu, cpu); +unsigned int tasks[NR_PSI_TASK_COUNTS ]; +u64 now, state_start;unsigned int seq; -u32 time, delta; +int s; +/* Snapshot a coherent view of the CPU state */ do { seq =read_seqcount_begin (&groupc->seq); + now =cpu_clock (cpu); +memcpy (times, groupc->times,sizeof (groupc->times)); +memcpy (tasks, groupc->tasks,sizeof (groupc->tasks)); + state_start = groupc->state_start; + }while (read_seqcount_retry (&groupc->seq, seq)); - time = groupc->times[state]; +/* Calculate state time deltas against the previous snapshot */ +for (s =0 ; s <NR_PSI_STATES ; s++) { +u32 delta;/* * In addition to already concluded states, we also * incorporate currently active states on the CPU,
@@ -217,14 +226,14 @@ static u32 get_recent_time(struct psi_group *group, int cpu,* (u32) and our reported pressure close to what's * actually happening. */ -if (test_state (groupc->tasks, state)) - time +=cpu_clock (cpu) - groupc->state_start; - }while (read_seqcount_retry (&groupc->seq, seq)); +if (test_state (tasks, s)) + times[s] += now - state_start; - delta = time - groupc->times_prev[state]; - groupc->times_prev[state] = time; + delta = times[s] - groupc->times_prev[s]; + groupc->times_prev[s] = times[s]; -return delta; + times[s] = delta; + } } static void calc_avgs(unsigned long avg[3], int missed_periods,
@@ -267,18 +276,16 @@ static bool update_stats(struct psi_group *group)* loading, or even entirely idle CPUs. */ for_each_possible_cpu (cpu) { +u32 times[NR_PSI_STATES ];u32 nonidle; - nonidle =get_recent_time (group, cpu,PSI_NONIDLE ); - nonidle =nsecs_to_jiffies (nonidle); - nonidle_total += nonidle; +get_recent_times (group, cpu, times); -for (s =0 ; s <PSI_NONIDLE ; s++) { -u32 delta; + nonidle =nsecs_to_jiffies (times[PSI_NONIDLE ]); + nonidle_total += nonidle; - delta =get_recent_time (group, cpu, s); - deltas[s] += (u64 )delta * nonidle; - } +for (s =0 ; s <PSI_NONIDLE ; s++) + deltas[s] += (u64 )times[s] * nonidle; }/*
Peter Zijlstra
maintainerRe: [PATCH 8/9] psi: pressure stall information for CPU, memory, and IO
In reply to Johannes WeinerQuoted 19 lines · Johannes Weiner
kernel/sched/psi.c new file +19
-0+static void psi_clock (struct work_struct *work) +{ +struct delayed_work *dwork; +struct psi_group *group; +bool nonidle; + + dwork =to_delayed_work (work); + group = container_of(dwork, struct psi_group, clock_work); + +/* +* If there is task activity, periodically fold the per-cpu +* times and feed samples into the running averages. If things +* are idle and there is no data to process, stop the clock. +* Once restarted, we'll catch up the running averages in one +* go - see calc_avgs() and missed_periods. +*/ + + nonidle =update_stats (group); +
kernel/sched/psi.c line 378
Quoted 10 lines
kernel/sched/psi.c new file +10
-0+if (nonidle) { +unsigned long delay =0 ; +u64 now; + + now =sched_clock (); +if (group->next_update > now) + delay =nsecs_to_jiffies (group->next_update - now) +1 ; +schedule_delayed_work (dwork, delay); + } +}
Just a little nit; I would expect a function called *clock() to return a time.
Johannes Weiner
maintainerRe: [PATCH 8/9] psi: pressure stall information for CPU, memory, and IO
Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Johannes Weiner
(quoted diff) +24
-0+static void psi_clock(struct work_struct *work) +{ + struct delayed_work *dwork; + struct psi_group *group; + bool nonidle; + + dwork = to_delayed_work(work); + group = container_of(dwork, struct psi_group, clock_work); + + /* + * If there is task activity, periodically fold the per-cpu + * times and feed samples into the running averages. If things + * are idle and there is no data to process, stop the clock. + * Once restarted, we'll catch up the running averages in one + * go - see calc_avgs() and missed_periods. + */ + + nonidle = update_stats(group); + + if (nonidle) { + unsigned long delay = 0; + u64 now; + + now = sched_clock();
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ if (group->next_update > now) + delay = nsecs_to_jiffies(group->next_update - now) + 1; + schedule_delayed_work(dwork, delay); + } +}Just a little nit; I would expect a function called
*clock()to return a time.
Fair enough, let's rename this. How about this on top?
kernel/sched/psi.c +3 -3
@@ -154,7 +154,7 @@ static struct psi_group psi_system = { .pcpu = &system_group_pcpu, }; -static void psi_clock (struct work_struct *work); +static void psi_update_work (struct work_struct *work);static void group_init (struct psi_group *group) {
@@ -163,7 +163,7 @@ static void group_init(struct psi_group *group)for_each_possible_cpu (cpu)seqcount_init (&per_cpu_ptr (group->pcpu, cpu)->seq); group->next_update =sched_clock () + psi_period; -INIT_DELAYED_WORK (&group->clock_work, psi_clock); +INIT_DELAYED_WORK (&group->clock_work, psi_update_work);mutex_init (&group->stat_lock); }
@@ -347,7 +347,7 @@ static bool update_stats(struct psi_group *group)return nonidle_total; } -static void psi_clock (struct work_struct *work) +static void psi_update_work (struct work_struct *work) {struct delayed_work *dwork;struct psi_group *group;
Johannes Weiner
maintainerPATCH 9/9 psi: cgroup support
In reply to Johannes WeinerOn a system that executes multiple cgrouped jobs and independent workloads, we don't just care about the health of the overall system, but also that of individual jobs, so that we can ensure individual job health, fairness between jobs, or prioritize some jobs over others.
This patch implements pressure stall tracking for cgroups. In kernels with CONFIG_PSI=y, cgroup2 groups will have cpu.pressure, memory.pressure, and io.pressure files that track aggregate pressure stall times for only the tasks inside the cgroup.
v3:
- fix copy-paste indentation screwups
v4:
- propagate psi_disabled checks outward
- factor out iterate_groups()
--- Documentation/accounting/psi.txt | 9 ++ Documentation/admin-guide/cgroup-v2.rst | 18 ++++ include/linux/cgroup-defs.h | 4 + include/linux/cgroup.h | 15 +++ include/linux/psi.h | 25 +++++ init/Kconfig | 4 + kernel/cgroup/cgroup.c | 45 ++++++++- kernel/sched/psi.c | 118 ++++++++++++++++++++++-- 8 files changed, 228 insertions(+), 10 deletions(-)
Documentation/accounting/psi.txt +9 -0
@@ -62,3 +62,12 @@ well as medium and long term trends. The total absolute stall time is tracked and exported as well, to allow detection of latency spikes which wouldn't necessarily make a dent in the time averages, or to average trends over custom time frames. + +Cgroup2 interface +================= + +In a system with a CONFIG_CGROUP=y kernel and the cgroup2 filesystem +mounted, pressure stall information is also tracked for tasks grouped +into cgroups. Each subdirectory in the cgroupfs mountpoint contains +cpu.pressure, memory.pressure, and io.pressure files; the format is +the same as the /proc/pressure/ files.
Documentation/admin-guide/cgroup-v2.rst +18 -0
@@ -963,6 +963,12 @@ All time durations are in microseconds. $PERIOD duration. "max" for $MAX indicates no limit. If only one number is written, $MAX is updated. + cpu.pressure + A read-only nested-key file which exists on non-root cgroups. + + Shows pressure stall information for CPU. See + Documentation/accounting/psi.txt for details. + Memory ------
@@ -1250,6 +1256,12 @@ PAGE_SIZE multiple when read back. higher than the limit for an extended period of time. This reduces the impact on the workload and memory management. + memory.pressure + A read-only nested-key file which exists on non-root cgroups. + + Shows pressure stall information for memory. See + Documentation/accounting/psi.txt for details. + Usage Guidelines ~~~~~~~~~~~~~~~~
@@ -1385,6 +1397,12 @@ IO Interface Files 8:16 rbps=2097152 wbps=max riops=max wiops=max + io.pressure + A read-only nested-key file which exists on non-root cgroups. + + Shows pressure stall information for IO. See + Documentation/accounting/psi.txt for details. + Writeback ~~~~~~~~~
include/linux/cgroup-defs.h +4 -0
@@ -20,6 +20,7 @@#include <linux/u64_stats_sync.h> #include <linux/workqueue.h> #include <linux/bpf-cgroup.h> +#include <linux/psi_types.h> #ifdef CONFIG_CGROUPS
@@ -435,6 +436,9 @@ struct cgroup {/* used to schedule release agent */ struct work_struct release_agent_work; +/* used to track pressure stalls */ +struct psi_group psi; +/* used to store eBPF programs */ struct cgroup_bpf bpf;
include/linux/cgroup.h +15 -0
@@ -627,6 +627,11 @@ static inline void pr_cont_cgroup_path(struct cgroup *cgrp)pr_cont_kernfs_path (cgrp->kn); } +static inline struct psi_group *cgroup_psi (struct cgroup *cgrp) +{ +return &cgrp->psi; +} +static inline void cgroup_init_kthreadd (void ) {/*
@@ -680,6 +685,16 @@ static inline union kernfs_node_id *cgroup_get_kernfs_id(struct cgroup *cgrp)return NULL ; } +static inline struct cgroup *cgroup_parent (struct cgroup *cgrp) +{ +return NULL ; +} + +static inline struct psi_group *cgroup_psi (struct cgroup *cgrp) +{ +return NULL ; +} +static inline bool task_under_cgroup_hierarchy (struct task_struct *task,struct cgroup *ancestor) {
include/linux/psi.h +25 -0
@@ -4,6 +4,9 @@#include <linux/psi_types.h> #include <linux/sched.h> +struct seq_file ; +struct css_set ; +#ifdef CONFIG_PSI extern bool psi_disabled;
@@ -16,6 +19,14 @@ void psi_memstall_tick(struct task_struct *task, int cpu);void psi_memstall_enter (unsigned long *flags);void psi_memstall_leave (unsigned long *flags); +int psi_show (struct seq_file *s,struct psi_group *group,enum psi_res res); + +#ifdef CONFIG_CGROUPS +int psi_cgroup_alloc (struct cgroup *cgrp); +void psi_cgroup_free (struct cgroup *cgrp); +void cgroup_move_task (struct task_struct *p,struct css_set *to); +#endif +#else /* CONFIG_PSI */ static inline void psi_init (void ) {}
@@ -23,6 +34,20 @@ static inline void psi_init(void) {}static inline void psi_memstall_enter (unsigned long *flags) {}static inline void psi_memstall_leave (unsigned long *flags) {} +#ifdef CONFIG_CGROUPS +static inline int psi_cgroup_alloc (struct cgroup *cgrp) +{ +return 0 ; +} +static inline void psi_cgroup_free (struct cgroup *cgrp) +{ +} +static inline void cgroup_move_task (struct task_struct *p,struct css_set *to) +{ +rcu_assign_pointer (p->cgroups, to); +} +#endif +#endif /* CONFIG_PSI */ #endif /* _LINUX_PSI_H */
init/Kconfig +4 -0
@@ -466,6 +466,10 @@ config PSI the share of walltime in which some or all tasks in the system are delayed due to contention of the respective resource. + In kernels with cgroup support, cgroups (cgroup2 only) will + have cpu.pressure, memory.pressure, and io.pressure files, + which aggregate pressure stalls for the grouped tasks only. + For more details see Documentation/accounting/psi.txt. Say N if unsure.
kernel/cgroup/cgroup.c +43 -2
90 lines not on this page · raw message
kernel/sched/psi.c +110 -8
144 lines not on this page · raw message
Signature · 1 line
2.18.0
Johannes Weiner
maintainerQuoted 4 lines · Johannes Weiner
This version 4 of the PSI series incorporates feedback from Peter and fixes two races in the lockless aggregator that Suren found in his testing and which caused the sample calculation to sometimes underflow and record bogusly large samples; details at the bottom of this email.
Peter, do the changes from v3 look sane to you?
If there aren't any further objections, I was hoping we could get this lined up for 4.20.
Quoted 4 lines · Johannes Weiner
Peter, do the changes from v3 look sane to you?
If there aren't any further objections, I was hoping we could get this lined up for 4.20.
That would be excellent. I just retested the latest version at
http://git.cmpxchg.org/cgit.cgi/linux-psi.git (Linux 4.18) and the results are great.
Test setup:
Endless OS
GeminiLake N4200 low end laptop
2GB RAM
swap (and zram swap) disabled
Baseline test: open a handful of large-ish apps and several website tabs in Google Chrome.
Results: after a couple of minutes, system is excessively thrashing, mouse cursor can barely be moved, UI is not responding to mouse clicks, so it's impractical to recover from this situation as an ordinary user
Add my simple killer:
https://gist.github.com/dsd/a8988bf0b81a6163475988120fe8d9cd
Results: when the thrashing causes the UI to become sluggish, the killer steps in and kills something (usually a chrome tab), and the system remains usable. I repeatedly opened more apps and more websites over a 15 minute period but I wasn't able to get the system to a point of UI unresponsiveness.
Thanks,
Daniel
Peter Zijlstra
maintainerQuoted 7 lines · Johannes Weiner
Quoted 4 lines · Johannes Weiner
This version 4 of the PSI series incorporates feedback from Peter and fixes two races in the lockless aggregator that Suren found in his testing and which caused the sample calculation to sometimes underflow and record bogusly large samples; details at the bottom of this email.
Peter, do the changes from v3 look sane to you?
I'll go have a look.
Peter Zijlstra
maintainerQuoted 10 lines · Johannes Weiner
Quoted 4 lines · Johannes Weiner
This version 4 of the PSI series incorporates feedback from Peter and fixes two races in the lockless aggregator that Suren found in his testing and which caused the sample calculation to sometimes underflow and record bogusly large samples; details at the bottom of this email.
Peter, do the changes from v3 look sane to you?
If there aren't any further objections, I was hoping we could get this lined up for 4.20.
I suppose it looks ok, there's a few small nits, but nothing big.
I still hate psi_ttwu_dequeue(), but I don't really know what to about that.
So yeah, grudingly acked. Did you want me to pick this up through the scheduler tree since most of this lives there?
Johannes Weiner
maintainerQuoted 2 lines · Peter Zijlstra
So yeah, grudingly acked. Did you want me to pick this up through the scheduler tree since most of this lives there?
Thanks for the ack.
As for routing it, I'll leave that decision to you and Andrew. It touches stuff all over, so it could result in quite a few conflicts between trees (although I don't expect any of them to be non-trivial).
Thanks for the new patchset! Backported to 4.9 and retested on ARMv8 8 code system running Android. Signals behave as expected reacting to memory pressure, no jumps in "total" counters that would indicate an overflow/underflow issues. Nicely done!
Quoted 9 lines · Johannes Weiner
Quoted 2 lines · Peter Zijlstra
So yeah, grudingly acked. Did you want me to pick this up through the scheduler tree since most of this lives there?
Thanks for the ack.
As for routing it, I'll leave that decision to you and Andrew. It touches stuff all over, so it could result in quite a few conflicts between trees (although I don't expect any of them to be non-trivial).
Hi Suren
Quoted 4 lines · Suren Baghdasaryan
Thanks for the new patchset! Backported to 4.9 and retested on ARMv8 8 code system running Android. Signals behave as expected reacting to memory pressure, no jumps in "total" counters that would indicate an overflow/underflow issues. Nicely done!
Can you share your Linux v4.9 psi backport somewhere?
Thanks
Daniel
Suren Baghdasaryan
maintainerHi Daniel,
Quoted 10 lines · Daniel Drake
Hi Suren
Quoted 4 lines · Suren Baghdasaryan
Thanks for the new patchset! Backported to 4.9 and retested on ARMv8 8 code system running Android. Signals behave as expected reacting to memory pressure, no jumps in "total" counters that would indicate an overflow/underflow issues. Nicely done!
Can you share your Linux v4.9 psi backport somewhere?
Absolutely. Let me figure out what's the best way to do share that and make sure they apply cleanly on official 4.9 (I was using vendor's tree for testing). Will need a day or so to get this done. In case you need them sooner, there were several "prerequisite" patches that I had to backport to make PSI backporting easier/possible. Following is the list as shown by "git log --oneline":
PSI patches:
ef94c067f360 psi: cgroup support60081a7aeb0b psi: pressure stall information for CPU, memory, and IO acd2a16497e9 sched: introduce this_rq_lock_irq()f30268c29309 sched: sched.h: make rq locking and clock functions available in stats.ha2fd1c94b743 sched: loadavg: make calc_load_n() public 32a74dec4967 sched: loadavg: consolidate LOAD_INT, LOAD_FRAC, CALC_LOAD 8e3991dd1a73 delayacct: track delays from thrashing cache pages 4ae940e7e6ff mm: workingset: tell cache transitions from workingset thrashing e9ccd63399e0 mm: workingset: don't drop refault information prematurely
Prerequisites:
b5a58c778c54 workqueue: make workqueue available early during boot ae5f39ee13b5 sched/core: Add wrappers for lockdep_(un)pin_lock() 7276f98a72c1 sched/headers, delayacct: Move the 'struct task_delay_info' definition from <linux/sched.h> to <linux/delayacct.h>287318d13688 mm: add PageWaiters indicating tasks are waiting for a page bit edfa64560aaa sched/headers: Remove <linux/sched.h> from <linux/sched/loadavg.h> f6b6ba853959 sched/headers: Move loadavg related definitions from <linux/sched.h> to <linux/sched/loadavg.h>395b0a9f7aae sched/headers: Prepare for new header dependencies before moving code to <linux/sched/loadavg.h>
PSI patches needed some adjustments but nothing really major.
Quoted 2 lines
Thanks
Daniel
Thanks,
Suren.
Suren Baghdasaryan
maintainerI emailed Daniel 4.9 backport patches. Unfortunately that seems to be the easiest way to share them. If anyone else is interested in them please email me directly.
Thanks,
Suren.
Quoted 56 lines · Suren Baghdasaryan
Hi Daniel,
Quoted 10 lines · Daniel Drake
Hi Suren
Quoted 4 lines · Suren Baghdasaryan
Thanks for the new patchset! Backported to 4.9 and retested on ARMv8 8 code system running Android. Signals behave as expected reacting to memory pressure, no jumps in "total" counters that would indicate an overflow/underflow issues. Nicely done!
Can you share your Linux v4.9 psi backport somewhere?
Absolutely. Let me figure out what's the best way to do share that and make sure they apply cleanly on official 4.9 (I was using vendor's tree for testing). Will need a day or so to get this done. In case you need them sooner, there were several "prerequisite" patches that I had to backport to make PSI backporting easier/possible. Following is the list as shown by "git log --oneline":
PSI patches:
ef94c067f360psi: cgroup support60081a7aeb0bpsi: pressure stall information for CPU, memory, and IOacd2a16497e9sched: introducethis_rq_lock_irq()f30268c29309sched:sched.h: make rq locking and clock functions available instats.ha2fd1c94b743sched: loadavg: makecalc_load_n()public32a74dec4967sched: loadavg: consolidateLOAD_INT,LOAD_FRAC,CALC_LOAD8e3991dd1a73delayacct: track delays from thrashing cache pages4ae940e7e6ffmm: workingset: tell cache transitions from workingset thrashinge9ccd63399e0mm: workingset: don't drop refault information prematurelyPrerequisites:
b5a58c778c54workqueue: make workqueue available early during bootae5f39ee13b5sched/core: Add wrappers for lockdep_(un)pin_lock()7276f98a72c1sched/headers, delayacct: Move the 'structtask_delay_info' definition from <linux/sched.h> to <linux/delayacct.h>287318d13688mm: add PageWaiters indicating tasks are waiting for a page bitedfa64560aaasched/headers: Remove <linux/sched.h> from <linux/sched/loadavg.h>f6b6ba853959sched/headers: Move loadavg related definitions from <linux/sched.h> to <linux/sched/loadavg.h>395b0a9f7aaesched/headers: Prepare for new header dependencies before moving code to <linux/sched/loadavg.h>PSI patches needed some adjustments but nothing really major.
Quoted 2 lines
Thanks
DanielThanks,
Suren.
Will it be part of the backport to 4.9 google android or is it for test only? I guess that this patch is to big for the LTS tree.
Quoted 16 lines · Suren Baghdasaryan
Thanks for the new patchset! Backported to 4.9 and retested on ARMv8 8 code system running Android. Signals behave as expected reacting to memory pressure, no jumps in "total" counters that would indicate an overflow/underflow issues. Nicely done!
Tested-by Suren Baghdasaryan <[email protected]>Quoted 8 lines · Johannes Weiner
Quoted 2 lines · Peter Zijlstra
So yeah, grudingly acked. Did you want me to pick this up through the scheduler tree since most of this lives there?
Thanks for the ack.
As for routing it, I'll leave that decision to you and Andrew. It touches stuff all over, so it could result in quite a few conflicts between trees (although I don't expect any of them to be non-trivial).
Peter Zijlstra
maintainerA: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?
Suren Baghdasaryan
maintainerQuoted 1 line · peter enderborg
Will it be part of the backport to 4.9 google android or is it for test only?
Currently I'm testing these patches in tandem with PSI monitor that I'm developing and test results look good. If things go well and we start using PSI for Android I will try to upstream the backport. If upstream rejects it we will have to merge it into Android common kernel repo as a last resort. Hope this answers your question.
Quoted 21 lines
I guess that this patch is to big for the LTS tree.
Quoted 16 lines · Suren Baghdasaryan
Thanks for the new patchset! Backported to 4.9 and retested on ARMv8 8 code system running Android. Signals behave as expected reacting to memory pressure, no jumps in "total" counters that would indicate an overflow/underflow issues. Nicely done!
Tested-by Suren Baghdasaryan <[email protected]>Quoted 8 lines · Johannes Weiner
Quoted 2 lines · Peter Zijlstra
So yeah, grudingly acked. Did you want me to pick this up through the scheduler tree since most of this lives there?
Thanks for the ack.
As for routing it, I'll leave that decision to you and Andrew. It touches stuff all over, so it could result in quite a few conflicts between trees (although I don't expect any of them to be non-trivial).
Thanks,
Suren.
Andrew Morton
maintainerQuoted 4 lines · On Tue, 28 Aug 2018 13:22:49 -0400 Johannes Weiner <[email protected]> wrote:
This version 4 of the PSI series incorporates feedback from Peter and fixes two races in the lockless aggregator that Suren found in his testing and which caused the sample calculation to sometimes underflow and record bogusly large samples; details at the bottom of this email.
We've had very little in the way of review activity for the PSI patchset. According to the changelog tags, anyway.
Johannes Weiner
maintainerQuoted 9 lines · Andrew Morton
Quoted 4 lines · On Tue, 28 Aug 2018 13:22:49 -0400 Johannes Weiner <[email protected]> wrote:
This version 4 of the PSI series incorporates feedback from Peter and fixes two races in the lockless aggregator that Suren found in his testing and which caused the sample calculation to sometimes underflow and record bogusly large samples; details at the bottom of this email.
We've had very little in the way of review activity for the PSI patchset. According to the changelog tags, anyway.
Peter reviewed it quite extensively over all revisions, and acked the final version. Peter, can we add your acked-by or reviewed-by tag(s)?
The scheduler part accounts for 99% of the complexity in those patches. The mm bits, while somewhat sprawling, are mostly mechanical.
Quoted 8 lines · Johannes Weiner
Quoted 7 lines · Andrew Morton
Quoted 4 lines · On Tue, 28 Aug 2018 13:22:49 -0400 Johannes Weiner <[email protected]> wrote:
This version 4 of the PSI series incorporates feedback from Peter and fixes two races in the lockless aggregator that Suren found in his testing and which caused the sample calculation to sometimes underflow and record bogusly large samples; details at the bottom of this email.
Quoted 5 lines
Quoted 2 lines
We've had very little in the way of review activity for the PSI patchset. According to the changelog tags, anyway.
Peter reviewed it quite extensively over all revisions, and acked the final version. Peter, can we add your acked-by or reviewed-by tag(s)?
I don't really do reviewed by; but yes, I thought I already did; lemme find.
Quoted 2 lines
The scheduler part accounts for 99% of the complexity in those patches. The mm bits, while somewhat sprawling, are mostly mechanical.
Ah, I now see my mistake;
https://lkml.kernel.org/r/[email protected]
I forgot to include an actual tag therein. My bad.