NUMA Balancer Suite
Subsystems (9)
cover NUMA Balancer Suite
We have NUMA Balancing feature which always trying to move pages of a task to the node it executed more, while still got issues:
* page cache can't be handled
* no cgroup level balancing
Suppose we have a box with 4 cpu, two cgroup A & B each running 4 tasks, below scenery could be easily observed:
NODE0 | NODE1
|
CPU0 CPU1 | CPU2 CPU3task_A0 task_A1 | task_A2 task_A3task_B0 task_B1 | task_B2 task_B3
and usually with the equal memory consumption on each node, when tasks have similar behavior.
In this case numa balancing try to move pages of task_A0,1 & task_B0,1 to node 0, pages of task_A2,3 & task_B2,3 to node 1, but page cache will be located randomly, depends on the first read/write CPU location.
Let's suppose another scenery:
NODE0 | NODE1
|
CPU0 CPU1 | CPU2 CPU3task_A0 task_A1 | task_B0 task_B1task_A2 task_A3 | task_B2 task_B3
By switching the cpu & memory resources of task_A0,1 and task_B0,1, now workloads of cgroup A all on node 0, and cgroup B all on node 1, resource consumption are same but related tasks could share a closer cpu cache, while cache still randomly located.
Now what if the workloads generate lot's of page cache, and most of the memory accessing are page cache writing?
A page cache generated by task_A0 on NODE1 won't follow it to NODE0, but if task_A0 was already on NODE0 before it read/write files, caches will be there, so how to make sure this happen?
Usually we could solve this problem by binding workloads on a single node, if the cgroup A was binding to CPU0,1, then all the caches it generated will be on NODE0, the numa bonus will be maximum.
However, this require a very well administration on specified workloads, suppose in our cases if A & B are with a changing CPU requirement from 0% to 400%, then binding to a single node would be a bad idea.
So what we need is a way to detect memory topology on cgroup level, and try to migrate cpu/mem resources to the node with most of the caches there, as long as the resource is plenty on that node.
This patch set introduced:
* advanced per-cgroup numa statistic
* numa preferred node feature
* Numa Balancer module
Which helps to achieve an easy and flexible numa resource assignment, to gain numa bonus as much as possible.
Michael Wang (5):
numa: introduce per-cgroup numa balancing locality statistic
numa: append per-node execution info in memory.numa_stat
numa: introduce per-cgroup preferred numa node
numa: introduce numa balancer infrastructure
numa: numa balancer
drivers/Makefile | 1 + drivers/numa/Makefile | 1 + drivers/numa/numa_balancer.c | 715 +++++++++++++++++++++++++++++++++++++++++++ include/linux/memcontrol.h | 99 ++++++ include/linux/sched.h | 9 +- kernel/sched/debug.c | 8 + kernel/sched/fair.c | 41 +++ mm/huge_memory.c | 7 +- mm/memcontrol.c | 246 +++++++++++++++ mm/memory.c | 9 +- mm/mempolicy.c | 4 + 11 files changed, 1133 insertions(+), 7 deletions(-) create mode 100644 drivers/numa/Makefile create mode 100644 drivers/numa/numa_balancer.c
Signature · 1 line
2.14.4.44.g2045bb6
RFC PATCH 1/5 numa: introduce per-cgroup numa balancing locality, statistic
This patch introduced numa locality statistic, which try to imply the numa balancing efficiency per memory cgroup.
By doing 'cat /sys/fs/cgroup/memory/CGROUP_PATH/memory.numa_stat', we see new output line heading with 'locality', the format is:
locality 0~9% 10%~19% 20%~29% 30%~39% 40%~49% 50%~59% 60%~69% 70%~79%
80%~89% 90%~100%
interval means that on a task's last numa balancing, the percentage of accessing local pages, which we called numa balancing locality.
And the number means inside the cgroup, how many ticks we hit tasks with such locality are running, for example:
locality 7260278 54860 90493 209327 295801 462784 558897 667242
2786324 7399308
the 7260278 means that this cgroup have some tasks with 0~9% locality executed 7260278 ticks.
By monitoring the increment, we can check if the workload of a particular cgroup is doing well with numa, when most of the tasks are running with locality 0~9%, then something is wrong with your numa policy.
--- include/linux/memcontrol.h | 38 +++++++++++++++++++++++++++++++++++ include/linux/sched.h | 8 +++++++- kernel/sched/debug.c | 7 +++++++ kernel/sched/fair.c | 8 ++++++++ mm/huge_memory.c | 4 +--- mm/memcontrol.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++ mm/memory.c | 5 ++--- 7 files changed, 113 insertions(+), 7 deletions(-)
include/linux/memcontrol.h +38 -0
@@ -179,6 +179,27 @@ enum memcg_kmem_state { KMEM_ONLINE, }; +#ifdef CONFIG_NUMA_BALANCING + +enum memcg_numa_locality_interval { +PERCENT_0_9 , +PERCENT_10_19 , +PERCENT_20_29 , +PERCENT_30_39 , +PERCENT_40_49 , +PERCENT_50_59 , +PERCENT_60_69 , +PERCENT_70_79 , +PERCENT_80_89 , +PERCENT_90_100 , +NR_NL_INTERVAL , +}; + +struct memcg_stat_numa { +u64 locality[NR_NL_INTERVAL ]; +}; + +#endif #if defined(CONFIG_SMP) struct memcg_padding { char x[0];
@@ -311,6 +332,10 @@ struct mem_cgroup {struct list_head event_list;spinlock_t event_list_lock; +#ifdef CONFIG_NUMA_BALANCING +struct memcg_stat_numa __percpu *stat_numa; +#endif +struct mem_cgroup_per_node *nodeinfo[0 ];/* WARNING: nodeinfo must be the last member here */ };
@@ -818,6 +843,14 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm,void mem_cgroup_split_huge_fixup (struct page *head);#endif +#ifdef CONFIG_NUMA_BALANCING +extern void memcg_stat_numa_update (struct task_struct *p); +#else +static inline void memcg_stat_numa_update (struct task_struct *p) +{ +} +#endif +#else /* CONFIG_MEMCG */ #define MEM_CGROUP_ID_SHIFT 0
@@ -1156,6 +1189,11 @@ static inlinevoid count_memcg_event_mm (struct mm_struct *mm,enum vm_event_item idx) { } + +static inline void memcg_stat_numa_update (struct task_struct *p) +{ +} +#endif /* CONFIG_MEMCG */ /* idx can be of type enum memcg_stat_item or node_stat_item */
include/linux/sched.h +7 -1
@@ -1049,8 +1049,14 @@ struct task_struct {* scan window were remote/local or failed to migrate. The task scan * period is adapted based on the locality of the faults with different * weights depending on whether they were shared or private faults +* +* 0 -- remote faults +* 1 -- local faults +* 2 -- page migration failure +* 3 -- remote page accessing after page migration +* 4 -- local page accessing after page migration */ -unsigned long numa_faults_locality[3 ]; +unsigned long numa_faults_locality[5 ];unsigned long numa_pages_migrated;#endif /* CONFIG_NUMA_BALANCING */
kernel/sched/debug.c +7 -0
@@ -873,6 +873,13 @@ static void sched_show_numa(struct task_struct *p, struct seq_file *m)SEQ_printf (m,"current_node=%d, numa_group_id=%d\n" ,task_node (p),task_numa_group_id (p));show_numa_stats (p, m); +SEQ_printf (m,"faults_locality local=%lu remote=%lu failed=%lu " , + p->numa_faults_locality[1 ], + p->numa_faults_locality[0 ], + p->numa_faults_locality[2 ]); +SEQ_printf (m,"lhit=%lu rhit=%lu\n" , + p->numa_faults_locality[4 ], + p->numa_faults_locality[3 ]);mpol_put (pol);#endif }
kernel/sched/fair.c +8 -0
@@ -23,6 +23,7 @@#include "sched.h" #include <trace/events/sched.h> +#include <linux/memcontrol.h> /* * Targeted preemption latency for CPU-bound tasks:
@@ -2387,6 +2388,11 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)memset (p->numa_faults_locality,0 ,sizeof (p->numa_faults_locality)); } + p->numa_faults_locality[mem_node ==numa_node_id () ?4 :3 ] += pages; + +if (mem_node ==NUMA_NO_NODE ) +return ; +/* * First accesses are treated as private, otherwise consider accesses * to be private if the accessing pid has not changed
@@ -2604,6 +2610,8 @@ void task_tick_numa(struct rq *rq, struct task_struct *curr)if (!curr->mm || (curr->flags &PF_EXITING ) || work->next != work)return ; +memcg_stat_numa_update (curr); +/* * Using runtime rather than walltime has the dual advantage that * we (mostly) drive the selection from busy threads and that the
mm/huge_memory.c +1 -3
@@ -1621,9 +1621,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd)if (anon_vma)page_unlock_anon_vma_read (anon_vma); -if (page_nid !=NUMA_NO_NODE ) -task_numa_fault (last_cpupid, page_nid,HPAGE_PMD_NR , - flags); +task_numa_fault (last_cpupid, page_nid,HPAGE_PMD_NR , flags);return 0 ; }
mm/memcontrol.c +50 -0
78 lines not on this page · raw message
mm/memory.c +2 -3
@@ -3609,7 +3609,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)struct vm_area_struct *vma = vmf->vma;struct page *page =NULL ;int page_nid =NUMA_NO_NODE ; -int last_cpupid; +int last_cpupid =0 ;int target_nid;bool migrated = false;pte_t pte, old_pte;
@@ -3689,8 +3689,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) flags |=TNF_MIGRATE_FAIL ; out: -if (page_nid !=NUMA_NO_NODE ) -task_numa_fault (last_cpupid, page_nid,1 , flags); +task_numa_fault (last_cpupid, page_nid,1 , flags);return 0 ; }
Signature · 1 line
2.14.4.44.g2045bb6
Peter Zijlstra
maintainerRe: [RFC PATCH 1/5] numa: introduce per-cgroup numa balancing locality, statistic
Quoted 9 lines · 王贇
include/linux/memcontrol.h +9
-0+#ifdef CONFIG_NUMA_BALANCING + +enum memcg_numa_locality_interval { +PERCENT_0_9 , +PERCENT_10_19 , +PERCENT_20_29 , +PERCENT_30_39 , +PERCENT_40_49 , + PERCENT_50_59,
include/linux/memcontrol.h enum memcg_kmem_state { · line 200
Quoted 10 lines
include/linux/memcontrol.h +10
-0+PERCENT_60_69 , +PERCENT_70_79 , +PERCENT_80_89 , +PERCENT_90_100 , + NR_NL_INTERVAL, +}; + +struct memcg_stat_numa { +u64 locality[NR_NL_INTERVAL ]; +};
If you make that 8 it fits a single cacheline. Do you really need the additional resolution? If so, then 16 would be the next logical amount of buckets. 10 otoh makes no sense what so ever.
Re: [RFC PATCH 1/5] numa: introduce per-cgroup numa balancing locality, statistic
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · 王贇
include/linux/memcontrol.h +15
-0+#ifdef CONFIG_NUMA_BALANCING + +enum memcg_numa_locality_interval { +PERCENT_0_9 , +PERCENT_10_19 , +PERCENT_20_29 , +PERCENT_30_39 , +PERCENT_40_49 , +PERCENT_50_59 , +PERCENT_60_69 , +PERCENT_70_79 , +PERCENT_80_89 , +PERCENT_90_100 , +NR_NL_INTERVAL , +};
Quoted 8 lines
include/linux/memcontrol.h enum memcg_kmem_state { · line 200
Quoted 4 lines
include/linux/memcontrol.h +4
-0+ +struct memcg_stat_numa { +u64 locality[NR_NL_INTERVAL ]; +};If you make that 8 it fits a single cacheline. Do you really need the additional resolution? If so, then 16 would be the next logical amount of buckets. 10 otoh makes no sense what so ever.
Thanks for point out :-) not have to be 10, I think we can save first two and make it PERCENT_0_29, already wrong enough if it drops below 30% and it's helpless to know detail changes in this section.
Regards,
Michael Wang
Quoted 1 line
Peter Zijlstra
maintainerRe: [RFC PATCH 1/5] numa: introduce per-cgroup numa balancing locality, statistic
In reply to 王贇Quoted 7 lines · 王贇
include/linux/sched.h +5
-0+* 0 -- remote faults +* 1 -- local faults +* 2 -- page migration failure +* 3 -- remote page accessing after page migration +* 4 -- local page accessing after page migration
include/linux/sched.h void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags) · line 2394
Quoted 7 lines
include/linux/sched.h +4
-0@@ -2387,6 +2388,11 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)memset (p->numa_faults_locality,0 ,sizeof (p->numa_faults_locality)); } + p->numa_faults_locality[mem_node ==numa_node_id () ?4 :3 ] += pages; + +if (mem_node ==NUMA_NO_NODE ) +return ;
I'm confused on the meaning of 3 & 4. It says 'after page migration' but 'every' access if after 'a' migration. But even more confusingly, you even account it if we know the page has never been migrated.
So what are you really counting?
Re: [RFC PATCH 1/5] numa: introduce per-cgroup numa balancing locality, statistic
Quoted 20 lines · Peter Zijlstra
Quoted 7 lines · 王贇
include/linux/sched.h +5
-0+* 0 -- remote faults +* 1 -- local faults +* 2 -- page migration failure +* 3 -- remote page accessing after page migration +* 4 -- local page accessing after page migration include/linux/sched.h void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags) · line 2394
Quoted 7 lines
include/linux/sched.h +4
-0@@ -2387,6 +2388,11 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)memset (p->numa_faults_locality,0 ,sizeof (p->numa_faults_locality)); } + p->numa_faults_locality[mem_node ==numa_node_id () ?4 :3 ] += pages; + +if (mem_node ==NUMA_NO_NODE ) +return ;I'm confused on the meaning of 3 & 4. It says 'after page migration' but 'every' access if after 'a' migration. But even more confusingly, you even account it if we know the page has never been migrated.
Quoted 1 line
So what are you really counting
Here is try to get the times of a task accessing the local or remote pages, and on no migration cases we still account since it's also one time of accessing, remotely or locally.
'after page migration' means this accounting need to understand the real page position after PF, what ever migration failure or succeed, whatever page move to local/remote or untouched, we want to know the times a task accessed the page locally or remotely, on numa balancing period.
Regards,
Michael Wang
Quoted 1 line
Peter Zijlstra
maintainerRe: [RFC PATCH 1/5] numa: introduce per-cgroup numa balancing locality, statistic
In reply to 王贇kernel/sched/fair.c void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags) · line 2391
Quoted 1 line · 王贇
kernel/sched/fair.c +1
-0+ p->numa_faults_locality[mem_node ==numa_node_id () ?4 :3 ] += pages;
Possibly: 3 + !!(mem_node = numa_node_id()), generates better code.
Re: [RFC PATCH 1/5] numa: introduce per-cgroup numa balancing locality, statistic
Quoted 4 lines · Peter Zijlstra
kernel/sched/fair.c void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags) · line 2391
Quoted 1 line · 王贇
kernel/sched/fair.c +1
-0+ p->numa_faults_locality[mem_node ==numa_node_id () ?4 :3 ] += pages;Possibly: 3 + !!(mem_node =
numa_node_id()), generates better code.
Sounds good~ will apply in next version.
Regards,
Michael Wang
Quoted 1 line
Peter Zijlstra
maintainerRe: [RFC PATCH 1/5] numa: introduce per-cgroup numa balancing locality, statistic
Quoted 9 lines · 王贇
Quoted 4 lines · Peter Zijlstra
kernel/sched/fair.c void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags) · line 2391
Quoted 1 line · 王贇
kernel/sched/fair.c +1
-0+ p->numa_faults_locality[mem_node ==numa_node_id () ?4 :3 ] += pages;Possibly: 3 + !!(mem_node =
numa_node_id()), generates better code.Sounds good~ will apply in next version.
Well, check code gen first, of course.
RFC PATCH 2/5 numa: append per-node execution info in memory.numa_stat
In reply to 王贇This patch introduced numa execution information, to imply the numa efficiency.
By doing 'cat /sys/fs/cgroup/memory/CGROUP_PATH/memory.numa_stat', we see new output line heading with 'exectime', like:
exectime 24399843 27865444
which means the tasks of this cgroup executed 24399843 ticks on node 0, and 27865444 ticks on node 1.
Combined with the memory node info, we can estimate the numa efficiency, for example the memory.numa_stat show:
total=4613257 N0=6849 N1=3928327
...
exectime 24399843 27865444
there could be unmovable or cache pages on N1, then good locality could mean nothing since we are not tracing these type of pages, thus bind the workloads on the cpus of N1 worth a try, in order to achieve the maximum performance bonus.
--- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 13 +++++++++++++ 2 files changed, 14 insertions(+)
include/linux/memcontrol.h +1 -0
@@ -197,6 +197,7 @@ enum memcg_numa_locality_interval {struct memcg_stat_numa {u64 locality[NR_NL_INTERVAL ]; +u64 exectime; };#endif
mm/memcontrol.c +13 -0
@@ -3409,6 +3409,18 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) seq_printf(m, " %llu", sum); }seq_putc (m,'\n' ); + +seq_puts (m,"exectime" ); +for_each_online_node (nr) { +int cpu; +u64 sum =0 ; + +for_each_cpu (cpu,cpumask_of_node (nr)) + sum +=per_cpu (memcg->stat_numa->exectime, cpu); + +seq_printf (m," %llu" , sum); + } +seq_putc (m,'\n' );#endif return 0 ;
@@ -3437,6 +3449,7 @@ void memcg_stat_numa_update(struct task_struct *p) memcg =mem_cgroup_from_task (p);if (idx !=-1 )this_cpu_inc (memcg->stat_numa->locality[idx]); +this_cpu_inc (memcg->stat_numa->exectime); rcu_read_unlock(); }#endif
Signature · 1 line
2.14.4.44.g2045bb6
Peter Zijlstra
maintainerRe: [RFC PATCH 2/5] numa: append per-node execution info in memory.numa_stat
Quoted 10 lines · 王贇
This patch introduced numa execution information, to imply the numa efficiency.
By doing 'cat
/sys/fs/cgroup/memory/CGROUP_PATH/memory.numa_stat', we see new output line heading with 'exectime', like:exectime 24399843 27865444
which means the tasks of this cgroup executed 24399843 ticks on node 0, and 27865444 ticks on node 1.
I think we stopped reporting time in HZ to userspace a long long time ago. Please don't do that.
Re: [RFC PATCH 2/5] numa: append per-node execution info in memory.numa_stat
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · 王贇
This patch introduced numa execution information, to imply the numa efficiency.
By doing 'cat
/sys/fs/cgroup/memory/CGROUP_PATH/memory.numa_stat', we see new output line heading with 'exectime', like:
Quoted 7 lines
Quoted 4 lines
exectime 24399843 27865444
which means the tasks of this cgroup executed 24399843 ticks on node 0, and 27865444 ticks on node 1.
I think we stopped reporting time in HZ to userspace a long long time ago. Please don't do that.
Ah I see, let's make it us maybe?
Regards,
Michael Wang
Quoted 1 line
Peter Zijlstra
maintainerRe: [RFC PATCH 2/5] numa: append per-node execution info in memory.numa_stat
Quoted 12 lines · 王贇
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · 王贇
This patch introduced numa execution information, to imply the numa efficiency.
By doing 'cat
/sys/fs/cgroup/memory/CGROUP_PATH/memory.numa_stat', we see new output line heading with 'exectime', like:exectime 24399843 27865444
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
which means the tasks of this cgroup executed 24399843 ticks on node 0, and 27865444 ticks on node 1.
I think we stopped reporting time in HZ to userspace a long long time ago. Please don't do that.
Ah I see, let's make it us maybe?
ms might be best I think.
Re: [RFC PATCH 2/5] numa: append per-node execution info in memory.numa_stat
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · 王贇
Quoted 12 lines · Peter Zijlstra
Quoted 10 lines · 王贇
This patch introduced numa execution information, to imply the numa efficiency.
By doing 'cat
/sys/fs/cgroup/memory/CGROUP_PATH/memory.numa_stat', we see new output line heading with 'exectime', like:exectime 24399843 27865444
which means the tasks of this cgroup executed 24399843 ticks on node 0, and 27865444 ticks on node 1.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
I think we stopped reporting time in HZ to userspace a long long time ago. Please don't do that.
Ah I see, let's make it us maybe?
ms might be best I think.
Will be in next version.
Regards,
Michael Wang
Quoted 1 line
RFC PATCH 3/5 numa: introduce per-cgroup preferred numa node
In reply to 王贇This patch add a new entry 'numa_preferred' for each memory cgroup, by which we can now override the memory policy of the tasks inside a particular cgroup, combined with numa balancing, we now be able to migrate the workloads of a cgroup to the specified numa node, in gentle way.
The load balancing and numa prefer against each other on CPU locations, which lead into the situation that although a particular node is capable enough to hold all the workloads, tasks will still spread.
In order to acquire the numa benifit in this situation, load balancing should respect the prefer decision as long as the balancing won't be broken.
This patch try to forbid workloads leave memcg preferred node, when and only when numa preferred node configured, in case if load balancing can't find other tasks to move and keep failing, we will then giveup and allow the migration to happen.
--- include/linux/memcontrol.h | 34 +++++++++++++++++++ include/linux/sched.h | 1 + kernel/sched/debug.c | 1 + kernel/sched/fair.c | 33 +++++++++++++++++++ mm/huge_memory.c | 3 ++ mm/memcontrol.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++ mm/memory.c | 4 +++ mm/mempolicy.c | 4 +++ 8 files changed, 162 insertions(+)
include/linux/memcontrol.h +34 -0
@@ -335,6 +335,8 @@ struct mem_cgroup {#ifdef CONFIG_NUMA_BALANCING struct memcg_stat_numa __percpu *stat_numa; +s64 numa_preferred; +struct mutex numa_mutex;#endif struct mem_cgroup_per_node *nodeinfo[0 ];
@@ -846,10 +848,26 @@ void mem_cgroup_split_huge_fixup(struct page *head);#ifdef CONFIG_NUMA_BALANCING extern void memcg_stat_numa_update (struct task_struct *p); +extern int memcg_migrate_prep (int target_nid,int page_nid); +extern int memcg_preferred_nid (struct task_struct *p,gfp_t gfp); +extern struct page *alloc_page_numa_preferred (gfp_t gfp,unsigned int order);#else static inline void memcg_stat_numa_update (struct task_struct *p) { } +static inline int memcg_migrate_prep (int target_nid,int page_nid) +{ +return target_nid; +} +static inline int memcg_preferred_nid (struct task_struct *p,gfp_t gfp) +{ +return -1 ; +} +static inline struct page *alloc_page_numa_preferred (gfp_t gfp, +unsigned int order) +{ +return NULL ; +}#endif #else /* CONFIG_MEMCG */
@@ -1195,6 +1213,22 @@ static inline void memcg_stat_numa_update(struct task_struct *p) { } +static inline int memcg_migrate_prep (int target_nid,int page_nid) +{ +return target_nid; +} + +static inline int memcg_preferred_nid (struct task_struct *p,gfp_t gfp) +{ +return -1 ; +} + +static inline struct page *alloc_page_numa_preferred (gfp_t gfp, +unsigned int order) +{ +return NULL ; +} +#endif /* CONFIG_MEMCG */ /* idx can be of type enum memcg_stat_item or node_stat_item */
include/linux/sched.h +1 -0
@@ -422,6 +422,7 @@ struct sched_statistics {u64 nr_migrations_cold;u64 nr_failed_migrations_affine;u64 nr_failed_migrations_running; +u64 nr_failed_migrations_memcg;u64 nr_failed_migrations_hot;u64 nr_forced_migrations;
kernel/sched/debug.c +1 -0
@@ -934,6 +934,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,P_SCHEDSTAT (se.statistics.nr_migrations_cold);P_SCHEDSTAT (se.statistics.nr_failed_migrations_affine);P_SCHEDSTAT (se.statistics.nr_failed_migrations_running); +P_SCHEDSTAT (se.statistics.nr_failed_migrations_memcg);P_SCHEDSTAT (se.statistics.nr_failed_migrations_hot);P_SCHEDSTAT (se.statistics.nr_forced_migrations);P_SCHEDSTAT (se.statistics.nr_wakeups);
kernel/sched/fair.c +33 -0
@@ -6701,6 +6701,10 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int sd_flag, int wake_f new_cpu =find_idlest_cpu (sd, p, cpu, prev_cpu, sd_flag); }else if (sd_flag &SD_BALANCE_WAKE ) {/* XXX always ? */ /* Fast path */ +int pnid =memcg_preferred_nid (p,0 ); + +if (pnid !=NUMA_NO_NODE && pnid !=cpu_to_node (new_cpu)) + new_cpu = prev_cpu; new_cpu =select_idle_sibling (p, prev_cpu, new_cpu);
@@ -7404,12 +7408,36 @@ static int migrate_degrades_locality(struct task_struct *p, struct lb_env *env)return dst_weight < src_weight; } +static inline bool memcg_migrate_allow (struct task_struct *p, +struct lb_env *env) +{ +int src_nid, dst_nid, pnid; + +/* failed too much could imply balancing broken, now be a good boy */ +if (env->sd->nr_balance_failed > env->sd->cache_nice_tries) +return true; + + src_nid =cpu_to_node (env->src_cpu); + dst_nid =cpu_to_node (env->dst_cpu); + + pnid =memcg_preferred_nid (p,0 ); +if (pnid != dst_nid && pnid == src_nid) +return false; + +return true; +}#else static inline int migrate_degrades_locality (struct task_struct *p,struct lb_env *env) {return -1 ; } + +static inline bool memcg_migrate_allow (struct task_struct *p, +struct lb_env *env) +{ +return true; +}#endif /*
@@ -7470,6 +7498,11 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)return 0 ; } +if (!memcg_migrate_allow (p, env)) { +schedstat_inc (p->se.statistics.nr_failed_migrations_memcg); +return 0 ; + } +/* * Aggressive migration if: * 1) destination numa is preferred
mm/huge_memory.c +3 -0
@@ -1523,6 +1523,9 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd)*/ page_locked =trylock_page (page); target_nid =mpol_misplaced (page, vma, haddr); + + target_nid =memcg_migrate_prep (target_nid, page_nid); +if (target_nid ==NUMA_NO_NODE ) {/* If the page was locked, there are no parallel migrations */ if (page_locked)
mm/memcontrol.c +82 -0
100 lines not on this page · raw message
mm/memory.c +4 -0
@@ -70,6 +70,7 @@#include <linux/dax.h> #include <linux/oom.h> #include <linux/numa.h> +#include <linux/memcontrol.h> #include <asm/io.h> #include <asm/mmu_context.h>
@@ -3675,6 +3676,9 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) target_nid =numa_migrate_prep (page, vma, vmf->address, page_nid, &flags);pte_unmap_unlock (vmf->pte, vmf->ptl); + + target_nid =memcg_migrate_prep (target_nid, page_nid); +if (target_nid ==NUMA_NO_NODE ) {put_page (page); goto out;
mm/mempolicy.c +4 -0
@@ -2031,6 +2031,10 @@ alloc_pages_vma(gfp_t gfp, int order, struct vm_area_struct *vma, pol =get_vma_policy (vma, addr); + page =alloc_page_numa_preferred (gfp, order); +if (page) + goto out; +if (pol->mode ==MPOL_INTERLEAVE ) {unsigned nid;
Signature · 1 line
2.14.4.44.g2045bb6
Peter Zijlstra
maintainerRe: [RFC PATCH 3/5] numa: introduce per-cgroup preferred numa node
Quoted 5 lines · 王贇
mm/mempolicy.c +0
-0
mm/mempolicy.c alloc_pages_vma(gfp_t gfp, int order, struct vm_area_struct *vma, · line 2040
Quoted 10 lines
mm/mempolicy.c +4
-0@@ -2031,6 +2031,10 @@ alloc_pages_vma(gfp_t gfp, int order, struct vm_area_struct *vma, pol =get_vma_policy (vma, addr); + page =alloc_page_numa_preferred (gfp, order); +if (page) + goto out; +if (pol->mode ==MPOL_INTERLEAVE ) {unsigned nid;
This I think is wrong, it overrides app specific mbind() requests.
Re: [RFC PATCH 3/5] numa: introduce per-cgroup preferred numa node
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · 王贇
mm/mempolicy.c +0
-0@@ -2031,6 +2031,10 @@ alloc_pages_vma(gfp_t gfp, int order, struct vm_area_struct *vma, pol =get_vma_policy (vma, addr);
Quoted 9 lines
mm/mempolicy.c alloc_pages_vma(gfp_t gfp, int order, struct vm_area_struct *vma, · line 2040
Quoted 7 lines
mm/mempolicy.c +4
-0+ page =alloc_page_numa_preferred (gfp, order); +if (page) + goto out; +if (pol->mode ==MPOL_INTERLEAVE ) {unsigned nid;This I think is wrong, it overrides app specific
mbind()requests.
The original concern is that we scared the user apps insider cgroup deal wrong with memory policy and do bad behavior, but now I agree that we should not override the policy, the admin will take the responsibility.
Regards,
Michael Wang
Quoted 1 line
RFC PATCH 4/5 numa: introduce numa balancer infrastructure
In reply to 王贇Now we have the way to estimate and adjust numa preferred node for each memcg, next problem is how to use them.
Usually one will bind workloads with cpuset.cpus, combined with cpuset.mems or maybe better the memory policy to achieve numa bonus, however in complicated scenery like combined type of workloads or cpushare way of isolation, this kind of administration could make one crazy, what we need is a way to gain numa bonus automatically, maybe not maximum but as much as possible.
This patch introduced basic API for kernel module to do numa adjustment, later coming the numa balancer module to use them and try to gain numa bonus as much as possible, automatically.
API including:
* numa preferred control
* memcg callback hook
* memcg per-node page number acquire
--- include/linux/memcontrol.h | 26 ++++++++++++ mm/memcontrol.c | 101 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 127 insertions(+)
include/linux/memcontrol.h +26 -0
@@ -200,6 +200,11 @@ struct memcg_stat_numa {u64 exectime; }; +struct memcg_callback { +void (*init)(struct mem_cgroup *memcg); +void (*exit)(struct mem_cgroup *memcg); +}; +#endif #if defined(CONFIG_SMP ) struct memcg_padding {
@@ -337,6 +342,8 @@ struct mem_cgroup {struct memcg_stat_numa __percpu *stat_numa;s64 numa_preferred;struct mutex numa_mutex; +void *numa_private; +struct list_head numa_list;#endif struct mem_cgroup_per_node *nodeinfo[0 ];
@@ -851,6 +858,10 @@ extern void memcg_stat_numa_update(struct task_struct *p);extern int memcg_migrate_prep (int target_nid,int page_nid);extern int memcg_preferred_nid (struct task_struct *p,gfp_t gfp);extern struct page *alloc_page_numa_preferred (gfp_t gfp,unsigned int order); +extern int register_memcg_callback (void *cb); +extern int unregister_memcg_callback (void *cb); +extern void config_numa_preferred (struct mem_cgroup *memcg,int nid); +extern u64 memcg_numa_pages (struct mem_cgroup *memcg,int nid,u32 mask);#else static inline void memcg_stat_numa_update (struct task_struct *p) {
@@ -868,6 +879,21 @@ static inline struct page *alloc_page_numa_preferred(gfp_t gfp, {return NULL ; } +static inline int register_memcg_callback (void *cb) +{ +return -EINVAL ; +} +static inline int unregister_memcg_callback (void *cb) +{ +return -EINVAL ; +} +static inline void config_numa_preferred (struct mem_cgroup *memcg,int nid) +{ +} +static inline u64 memcg_numa_pages (struct mem_cgroup *memcg,int nid,u32 mask) +{ +return 0 ; +}#endif #else /* CONFIG_MEMCG */
mm/memcontrol.c +101 -0
119 lines not on this page · raw message
Signature · 1 line
2.14.4.44.g2045bb6
RFC PATCH 5/5 numa: numa balancer
In reply to 王贇numa balancer is a module which will try to automatically adjust numa balancing stuff to gain numa bonus as much as possible.
For each memory cgroup, we process the work in two steps:
On stage 1 we check cgroup's exectime and memory topology to see if there could be a candidate for settled down, if we got one then move onto stage 2.
On stage 2 we try to settle down as much as possible by prefer the candidate node, if the node no longer suitable or locality keep downturn, we reset things and new round begin.
Decision made with find_candidate_nid(), should_prefer() and keep_prefer(), which try to pick a candidate node, see if allowed to prefer it and if keep doing the prefer.
Tested on the box with 96 cpus with sysbench-mysql-oltp_read_write testing, 4 mysqld instances created and attached to 4 cgroups, 4 sysbench instances then created and attached to corresponding cgroup to test the mysql with oltp_read_write script, average eps show:
origin balancer
4 instances each 12 threads 5241.08 5375.59 +2.50%
4 instances each 24 threads 7497.29 7820.73 +4.13%
4 instances each 36 threads 8985.44 9317.04 +3.55%
4 instances each 48 threads 9716.50 9982.60 +2.66%
Other benchmark liks dbench, pgbench, perf bench numa also tested, and with different parameters and number of instances/threads, most of the cases show bonus, some show acceptable regression, and some got no changes.
TODO:
* improve the logical to address the regression cases
* Find a way, maybe, to handle the page cache left on remote
* find more scenery which could gain benefit
--- drivers/Makefile | 1 + drivers/numa/Makefile | 1 + drivers/numa/numa_balancer.c | 715 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 717 insertions(+) create mode 100644 drivers/numa/Makefile create mode 100644 drivers/numa/numa_balancer.c
drivers/Makefile +1 -0
@@ -187,3 +187,4 @@ obj-$(CONFIG_UNISYS_VISORBUS) += visorbus/ obj-$(CONFIG_SIOX) += siox/ obj-$(CONFIG_GNSS) += gnss/ obj-$(CONFIG_INTERCONNECT) += interconnect/ +obj-$(CONFIG_NUMA_BALANCING) += numa/
drivers/numa/Makefile new file +1 -0
@@ -0,0 +1 @@ +obj-m += numa_balancer.o
drivers/numa/numa_balancer.c new file +715 -0
715 lines not on this page · raw message
Signature · 1 line
2.14.4.44.g2045bb6
Peter Zijlstra
maintainerRe: [RFC PATCH 5/5] numa: numa balancer
Quoted 38 lines · 王贇
numa balancer is a module which will try to automatically adjust numa balancing stuff to gain numa bonus as much as possible.
For each memory cgroup, we process the work in two steps:
On stage 1 we check cgroup's exectime and memory topology to see if there could be a candidate for settled down, if we got one then move onto stage 2.
On stage 2 we try to settle down as much as possible by prefer the candidate node, if the node no longer suitable or locality keep downturn, we reset things and new round begin.
Decision made with
find_candidate_nid(),should_prefer()andkeep_prefer(), which try to pick a candidate node, see if allowed to prefer it and if keep doing the prefer.Tested on the box with 96 cpus with sysbench-mysql-oltp_read_write testing, 4 mysqld instances created and attached to 4 cgroups, 4 sysbench instances then created and attached to corresponding cgroup to test the mysql with
oltp_read_writescript, average eps show:origin balancer
4 instances each 12 threads 5241.08 5375.59 +2.50%
4 instances each 24 threads 7497.29 7820.73 +4.13%
4 instances each 36 threads 8985.44 9317.04 +3.55%
4 instances each 48 threads 9716.50 9982.60 +2.66%Other benchmark liks dbench, pgbench, perf bench numa also tested, and with different parameters and number of instances/threads, most of the cases show bonus, some show acceptable regression, and some got no changes.
TODO:
* improve the logical to address the regression cases
* Find a way, maybe, to handle the page cache left on remote
* find more scenery which could gain benefit
Quoted 5 lines
Signed-off-by Michael Wang <[email protected]>--- drivers/Makefile | 1 + drivers/numa/Makefile | 1 + drivers/numa/numa_balancer.c | 715 +++++++++++++++++++++++++++++++++++++++++++
So I really think this is the wrong direction. Why introduce yet another balancer thingy and not extend the existing numa balancer with the additional information you got from the previous patches?
Also, this really should not be a module and not in drivers/
Re: [RFC PATCH 5/5] numa: numa balancer
On 2019/4/23 下午5:05, Peter Zijlstra wrote:
[snip]
Quoted 12 lines
Quoted 11 lines
TODO:
* improve the logical to address the regression cases
* Find a way, maybe, to handle the page cache left on remote
* find more scenery which could gain benefitSigned-off-by Michael Wang <[email protected]>--- drivers/Makefile | 1 + drivers/numa/Makefile | 1 + drivers/numa/numa_balancer.c | 715 +++++++++++++++++++++++++++++++++++++++++++
Quoted 5 lines
So I really think this is the wrong direction. Why introduce yet another balancer thingy and not extend the existing numa balancer with the additional information you got from the previous patches?
Also, this really should not be a module and not in drivers
The reason why we present the idea in the way of a module is that it's not suitable for all the situations, a module could be clean and easier for deploy on demands.
Besides, we assume someone may prefer to have their own logical on how to do the numa balancer, thus the module give them the way to DIY easily.
But there are no insist on the style, once the logical is mature enough, we can merge the idea into CFS, per-cgroup switch could be enough :-P
Regards,
Michael Wang
Quoted 1 line
Hi, Michael
I really want to know how could you fix the conflict between numa balancer and load balancer. Maybe you gained numa bonus by migrating some tasks to the node with most of the cache there, but, cpu load balance was break, so how to do it ?
Thanks
Wind
Quoted 103 lines · 王贇 <[email protected]> 于2019年4月22日周一 上午10:13写道:
We have NUMA Balancing feature which always trying to move pages of a task to the node it executed more, while still got issues:
* page cache can't be handled
* no cgroup level balancingSuppose we have a box with 4 cpu, two cgroup A & B each running 4 tasks, below scenery could be easily observed:
NODE0 | NODE1 | CPU0 CPU1 | CPU2 CPU3 task_A0 task_A1 | task_A2 task_A3 task_B0 task_B1 | task_B2 task_B3and usually with the equal memory consumption on each node, when tasks have similar behavior.
In this case numa balancing try to move pages of task_A0,1 & task_B0,1 to node 0,
pages of task_A2,3 & task_B2,3 to node 1, but page cache will be located randomly,
depends on the first read/write CPU location.Let's suppose another scenery:
NODE0 | NODE1 | CPU0 CPU1 | CPU2 CPU3 task_A0 task_A1 | task_B0 task_B1 task_A2 task_A3 | task_B2 task_B3By switching the cpu & memory resources of task_A0,1 and task_B0,1, now workloads
of cgroup A all on node 0, and cgroup B all on node 1, resource consumption are same
but related tasks could share a closer cpu cache, while cache still randomly located.Now what if the workloads generate lot's of page cache, and most of the memory
accessing are page cache writing?A page cache generated by
task_A0on NODE1 won't follow it to NODE0, but iftask_A0
was already on NODE0 before it read/write files, caches will be there, so how to
make sure this happen?Usually we could solve this problem by binding workloads on a single node, if the
cgroup A was binding to CPU0,1, then all the caches it generated will be on NODE0,
the numa bonus will be maximum.However, this require a very well administration on specified workloads, suppose in our
cases if A & B are with a changing CPU requirement from 0% to 400%, then binding to a
single node would be a bad idea.So what we need is a way to detect memory topology on cgroup level, and try to migrate
cpu/mem resources to the node with most of the caches there, as long as the resource
is plenty on that node.This patch set introduced:
* advanced per-cgroup numa statistic
* numa preferred node feature
* Numa Balancer moduleWhich helps to achieve an easy and flexible numa resource assignment, to gain numa bonus
as much as possible.Michael Wang (5):
numa: introduce per-cgroup numa balancing locality statistic
numa: append per-node execution info in memory.numa_stat
numa: introduce per-cgroup preferred numa node
numa: introduce numa balancer infrastructure
numa: numa balancerdrivers/Makefile | 1 + drivers/numa/Makefile | 1 + drivers/numa/numa_balancer.c | 715 +++++++++++++++++++++++++++++++++++++++++++ include/linux/memcontrol.h | 99 ++++++ include/linux/sched.h | 9 +- kernel/sched/debug.c | 8 + kernel/sched/fair.c | 41 +++ mm/huge_memory.c | 7 +- mm/memcontrol.c | 246 +++++++++++++++ mm/memory.c | 9 +- mm/mempolicy.c | 4 + 11 files changed, 1133 insertions(+), 7 deletions(-) create mode 100644 drivers/numa/Makefile create mode 100644 drivers/numa/numa_balancer.c--
2.14.4.44.g2045bb6
Quoted 2 lines · 禹舟键
Hi, Michael
I really want to know how could you fix the conflict between numa balancer and load balancer. Maybe you gained numa bonus by migrating some tasks to the node with most of the cache there, but, cpu load balance was break, so how to do it ?
The trick here is to allow migration when load balancing keep failing, which means no better tasks to move.
However, since the idea here is cgroup workloads scheduling, it could be hard to make sure load balanced, for example only two cgroup with different workloads and putting them to different node.
Thus why we make this a module, rather than changing the kernel logical, at this moment not every situation could gain benefit from numa balancer, but in some situations, balanced load can't bring benefit while numa balancer could.
Also we are improving the module to give it an overall sight, so it will know whether the decision is breaking the load balance, but this introduced big lock and more per cpu/node counters, we need more testing to know whether this is really helpful.
Anyway, if you have any scenery may could gain benefit, please take a try and let me know what's the problem is, we'll try to address them :-)
Regards,
Michael Wang
Quoted 93 lines
Thanks
Wind王贇 <[email protected] <mailto:[email protected]>> 于2019年4月22日周一 上午10:13写道:
We have NUMA Balancing feature which always trying to move pages of a task to the node it executed more, while still got issues:
* page cache can't be handled
* no cgroup level balancingSuppose we have a box with 4 cpu, two cgroup A & B each running 4 tasks, below scenery could be easily observed:
NODE0 | NODE1
|
CPU0 CPU1 | CPU2 CPU3 task_A0 task_A1 | task_A2 task_A3 task_B0 task_B1 | task_B2 task_B3and usually with the equal memory consumption on each node, when tasks have similar behavior.
In this case numa balancing try to move pages of task_A0,1 & task_B0,1 to node 0, pages of task_A2,3 & task_B2,3 to node 1, but page cache will be located randomly, depends on the first read/write CPU location.
Let's suppose another scenery:
NODE0 | NODE1
|
CPU0 CPU1 | CPU2 CPU3 task_A0 task_A1 | task_B0 task_B1 task_A2 task_A3 | task_B2 task_B3By switching the cpu & memory resources of task_A0,1 and task_B0,1, now workloads of cgroup A all on node 0, and cgroup B all on node 1, resource consumption are same but related tasks could share a closer cpu cache, while cache still randomly located.
Now what if the workloads generate lot's of page cache, and most of the memory accessing are page cache writing?
A page cache generated by
task_A0on NODE1 won't follow it to NODE0, but iftask_A0was already on NODE0 before it read/write files, caches will be there, so how to make sure this happen?Usually we could solve this problem by binding workloads on a single node, if the cgroup A was binding to CPU0,1, then all the caches it generated will be on NODE0, the numa bonus will be maximum.
However, this require a very well administration on specified workloads, suppose in our cases if A & B are with a changing CPU requirement from 0% to 400%, then binding to a single node would be a bad idea.
So what we need is a way to detect memory topology on cgroup level, and try to migrate cpu/mem resources to the node with most of the caches there, as long as the resource is plenty on that node.
This patch set introduced:
* advanced per-cgroup numa statistic
* numa preferred node feature
* Numa Balancer moduleWhich helps to achieve an easy and flexible numa resource assignment, to gain numa bonus as much as possible.
Michael Wang (5):
numa: introduce per-cgroup numa balancing locality statistic
numa: append per-node execution info in memory.numa_stat
numa: introduce per-cgroup preferred numa node
numa: introduce numa balancer infrastructure
numa: numa balancerdrivers/Makefile | 1 +
drivers/numa/Makefile | 1 +
drivers/numa/numa_balancer.c | 715 +++++++++++++++++++++++++++++++++++++++++++ include/linux/memcontrol.h | 99 ++++++
include/linux/sched.h | 9 +-
kernel/sched/debug.c | 8 +
kernel/sched/fair.c | 41 +++
mm/huge_memory.c | 7 +-
mm/memcontrol.c | 246 +++++++++++++++
mm/memory.c | 9 +-
mm/mempolicy.c | 4 +
11 files changed, 1133 insertions(+), 7 deletions(-)
create mode 100644drivers/numa/Makefile
create mode 100644drivers/numa/numa_balancer.c--
2.14.4.44.g2045bb6
cover per cpu cgroup numa suite
In reply to 王贇During our torturing on numa stuff, we found problems like:
* missing per-cgroup information about the per-node execution status
* missing per-cgroup information about the numa locality
That is when we have a cpu cgroup running with bunch of tasks, no good way to tell how it's tasks are dealing with numa.
The first two patches are trying to complete the missing pieces, but more problems appeared after monitoring these status:
* tasks not always running on the preferred numa node
* tasks from same cgroup running on different nodes
The task numa group handler will always check if tasks are sharing pages and try to pack them into a single numa group, so they will have chance to settle down on the same node, but this failed in some cases:
* workloads share page caches rather than share mappings
* workloads got too many wakeup across nodes
Since page caches are not traced by numa balancing, there are no way to realize such kind of relationship, and when there are too many wakeup, task will be drag from the preferred node and then migrate back by numa balancing, repeatedly.
Here the third patch try to address the first issue, we could now give hint to kernel about the relationship of tasks, and pack them into single numa group.
And the forth patch introduced numa cling, which try to address the wakup issue, now we try to make task stay on the preferred node on wakeup in fast path, in order to address the unbalancing risk, we monitoring the numa migration failure ratio, and pause numa cling when it reach the specified degree.
Michael Wang (4):
numa: introduce per-cgroup numa balancing locality statistic
numa: append per-node execution info in memory.numa_stat
numa: introduce numa group per task group
numa: introduce numa cling feature
include/linux/memcontrol.h | 37 ++++ include/linux/sched.h | 8 +- include/linux/sched/sysctl.h | 3 + kernel/sched/core.c | 37 ++++ kernel/sched/debug.c | 7 + kernel/sched/fair.c | 455 ++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 14 ++ kernel/sysctl.c | 9 + mm/memcontrol.c | 66 +++++++ 9 files changed, 628 insertions(+), 8 deletions(-)
Signature · 1 line
2.14.4.44.g2045bb6
PATCH 1/4 numa: introduce per-cgroup numa balancing locality, statistic
This patch introduced numa locality statistic, which try to imply the numa balancing efficiency per memory cgroup.
By doing 'cat /sys/fs/cgroup/memory/CGROUP_PATH/memory.numa_stat', we see new output line heading with 'locality', the format is:
locality 0%~29% 30%~39% 40%~49% 50%~59% 60%~69% 70%~79% 80%~89%
90%~100%
interval means that on a task's last numa balancing, the percentage of accessing local pages, which we called numa balancing locality.
And the number means inside the cgroup, how many micro seconds tasks with that locality are running, for example:
locality 15393 21259 13023 44461 21247 17012 28496 145402
the first number means that this cgroup have some tasks with 0~29% locality executed 15393 ms.
By monitoring the increment, we can check if the workload of a particular
cgroup is doing well with numa, when most of the tasks are running with locality 0~29%, then something is wrong with your numa policy.
--- include/linux/memcontrol.h | 36 +++++++++++++++++++++++++++++++ include/linux/sched.h | 8 ++++++- kernel/sched/debug.c | 7 ++++++ kernel/sched/fair.c | 9 ++++++++ mm/memcontrol.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 112 insertions(+), 1 deletion(-)
include/linux/memcontrol.h +36 -0
@@ -174,6 +174,25 @@ enum memcg_kmem_state { KMEM_ONLINE, }; +#ifdef CONFIG_NUMA_BALANCING + +enum memcg_numa_locality_interval { +PERCENT_0_29 , +PERCENT_30_39 , +PERCENT_40_49 , +PERCENT_50_59 , +PERCENT_60_69 , +PERCENT_70_79 , +PERCENT_80_89 , +PERCENT_90_100 , +NR_NL_INTERVAL , +}; + +struct memcg_stat_numa { +u64 locality[NR_NL_INTERVAL ]; +}; + +#endif #if defined(CONFIG_SMP) struct memcg_padding { char x[0];
@@ -313,6 +332,10 @@ struct mem_cgroup {struct list_head event_list;spinlock_t event_list_lock; +#ifdef CONFIG_NUMA_BALANCING +struct memcg_stat_numa __percpu *stat_numa; +#endif +struct mem_cgroup_per_node *nodeinfo[0 ];/* WARNING: nodeinfo must be the last member here */ };
@@ -795,6 +818,14 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm,void mem_cgroup_split_huge_fixup (struct page *head);#endif +#ifdef CONFIG_NUMA_BALANCING +extern void memcg_stat_numa_update (struct task_struct *p); +#else +static inline void memcg_stat_numa_update (struct task_struct *p) +{ +} +#endif +#else /* CONFIG_MEMCG */ #define MEM_CGROUP_ID_SHIFT 0
@@ -1131,6 +1162,11 @@ static inlinevoid count_memcg_event_mm (struct mm_struct *mm,enum vm_event_item idx) { } + +static inline void memcg_stat_numa_update (struct task_struct *p) +{ +} +#endif /* CONFIG_MEMCG */ /* idx can be of type enum memcg_stat_item or node_stat_item */
include/linux/sched.h +7 -1
@@ -1117,8 +1117,14 @@ struct task_struct {* scan window were remote/local or failed to migrate. The task scan * period is adapted based on the locality of the faults with different * weights depending on whether they were shared or private faults +* +* 0 -- remote faults +* 1 -- local faults +* 2 -- page migration failure +* 3 -- remote page accessing +* 4 -- local page accessing */ -unsigned long numa_faults_locality[3 ]; +unsigned long numa_faults_locality[5 ];unsigned long numa_pages_migrated;#endif /* CONFIG_NUMA_BALANCING */
kernel/sched/debug.c +7 -0
@@ -849,6 +849,13 @@ static void sched_show_numa(struct task_struct *p, struct seq_file *m)SEQ_printf (m,"current_node=%d, numa_group_id=%d\n" ,task_node (p),task_numa_group_id (p));show_numa_stats (p, m); +SEQ_printf (m,"faults_locality local=%lu remote=%lu failed=%lu " , + p->numa_faults_locality[1 ], + p->numa_faults_locality[0 ], + p->numa_faults_locality[2 ]); +SEQ_printf (m,"lhit=%lu rhit=%lu\n" , + p->numa_faults_locality[4 ], + p->numa_faults_locality[3 ]);mpol_put (pol);#endif }
kernel/sched/fair.c +9 -0
@@ -23,6 +23,7 @@#include "sched.h" #include <trace/events/sched.h> +#include <linux/memcontrol.h> /* * Targeted preemption latency for CPU-bound tasks:
@@ -2449,6 +2450,12 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags) p->numa_faults[task_faults_idx (NUMA_MEMBUF , mem_node, priv)] += pages; p->numa_faults[task_faults_idx (NUMA_CPUBUF , cpu_node, priv)] += pages; p->numa_faults_locality[local] += pages; +/* +* We want to have the real local/remote page access statistic +* here, so use 'mem_node' which is the real residential node of +* page after migrate_misplaced_page(). +*/ + p->numa_faults_locality[3 + !!(mem_node ==numa_node_id ())] += pages; }static void reset_ptenuma_scan (struct task_struct *p)
@@ -2625,6 +2632,8 @@ static void task_tick_numa(struct rq *rq, struct task_struct *curr)if (!curr->mm || (curr->flags &PF_EXITING ) || work->next != work)return ; +memcg_stat_numa_update (curr); +/* * Using runtime rather than walltime has the dual advantage that * we (mostly) drive the selection from busy threads and that the
mm/memcontrol.c +53 -0
81 lines not on this page · raw message
Signature · 1 line
2.14.4.44.g2045bb6
Peter Zijlstra
maintainerRe: [PATCH 1/4] numa: introduce per-cgroup numa balancing locality, statistic
include/linux/memcontrol.h enum memcg_kmem_state { · line 189
Quoted 13 lines · 王贇
include/linux/memcontrol.h +13
-0+#ifdef CONFIG_NUMA_BALANCING + +enum memcg_numa_locality_interval { +PERCENT_0_29 , +PERCENT_30_39 , +PERCENT_40_49 , +PERCENT_50_59 , +PERCENT_60_69 , +PERCENT_70_79 , +PERCENT_80_89 , +PERCENT_90_100 , +NR_NL_INTERVAL , +};
That's just daft; why not make 8 equal sized buckets.
include/linux/memcontrol.h enum memcg_kmem_state {
Quoted 6 lines
include/linux/memcontrol.h +5
-0+struct memcg_stat_numa { +u64 locality[NR_NL_INTERVAL ]; +}; +if (remote || local) { + idx = ((local *10 ) / (remote + local)) -2 ;
idx = (NR_NL_INTERVAL * local) / (remote + local);
Quoted 9 lines
mm/memcontrol.c +9
-0+ } + +rcu_read_lock (); + memcg =mem_cgroup_from_task (p); +if (idx !=-1 ) +this_cpu_inc (memcg->stat_numa->locality[idx]); +rcu_read_unlock (); +} +#endif
Re: [PATCH 1/4] numa: introduce per-cgroup numa balancing locality, statistic
Quoted 17 lines · Peter Zijlstra
include/linux/memcontrol.h enum memcg_kmem_state { · line 189
Quoted 13 lines · 王贇
include/linux/memcontrol.h +13
-0+#ifdef CONFIG_NUMA_BALANCING + +enum memcg_numa_locality_interval { +PERCENT_0_29 , +PERCENT_30_39 , +PERCENT_40_49 , +PERCENT_50_59 , +PERCENT_60_69 , +PERCENT_70_79 , +PERCENT_80_89 , +PERCENT_90_100 , +NR_NL_INTERVAL , +};That's just daft; why not make 8 equal sized buckets.
Quoted 8 lines
include/linux/memcontrol.h enum memcg_kmem_state {
Quoted 6 lines
include/linux/memcontrol.h +5
-0+struct memcg_stat_numa { +u64 locality[NR_NL_INTERVAL ]; +}; +if (remote || local) { + idx = ((local *10 ) / (remote + local)) -2 ;idx = (NR_NL_INTERVAL * local) / (remote + local);
Make sense, we actually want to observe the situation rather than the ratio itself, will be in next version.
Regards,
Michael Wang
Quoted 10 lines
Quoted 10 lines
mm/memcontrol.c +9
-0+ } + +rcu_read_lock (); + memcg =mem_cgroup_from_task (p); +if (idx !=-1 ) +this_cpu_inc (memcg->stat_numa->locality[idx]); +rcu_read_unlock (); +} +#endif
Peter Zijlstra
maintainerRe: [PATCH 1/4] numa: introduce per-cgroup numa balancing locality, statistic
In reply to 王贇Quoted 37 lines · 王贇
mm/memcontrol.c +29
-0@@ -3562,10 +3563,53 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) seq_putc(m, '\n'); } +#ifdef CONFIG_NUMA_BALANCING +seq_puts (m,"locality" ); +for (nr =0 ; nr <NR_NL_INTERVAL ; nr++) { +int cpu; +u64 sum =0 ; + +for_each_possible_cpu (cpu) + sum +=per_cpu (memcg->stat_numa->locality[nr], cpu); + +seq_printf (m," %u" ,jiffies_to_msecs (sum)); + } +seq_putc (m,'\n' ); +#endif +return 0 ; }#endif /* CONFIG_NUMA */ +#ifdef CONFIG_NUMA_BALANCING + +void memcg_stat_numa_update (struct task_struct *p) +{ +struct mem_cgroup *memcg; +unsigned long remote = p->numa_faults_locality[3 ]; +unsigned long local = p->numa_faults_locality[4 ]; +unsigned long idx =-1 ; + +if (mem_cgroup_disabled ()) +return ; + +if (remote || local) { + idx = ((local *10 ) / (remote + local)) -2 ; +/* 0~29% in one slot for cache align */
mm/memcontrol.c static int memcg_numa_stat_show(struct seq_file *m, void *v) · line 3608
Quoted 10 lines
mm/memcontrol.c +10
-0+if (idx <PERCENT_0_29 ) + idx =PERCENT_0_29 ; +else if (idx >=NR_NL_INTERVAL ) + idx =NR_NL_INTERVAL -1 ; + } + +rcu_read_lock (); + memcg =mem_cgroup_from_task (p); +if (idx !=-1 ) +this_cpu_inc (memcg->stat_numa->locality[idx]);
I thought cgroups were supposed to be hierarchical. That is, if we have:
R / \ A /\ B \ t1
Then our task t1 should be accounted to B (as you do), but also to A and R.
Quoted 3 lines
mm/memcontrol.c +3
-0+ rcu_read_unlock(); +} +#endif
Re: [PATCH 1/4] numa: introduce per-cgroup numa balancing locality, statistic
On 2019/7/11 下午9:47, Peter Zijlstra wrote:
[snip]
Quoted 15 lines
mm/memcontrol.c static int memcg_numa_stat_show(struct seq_file *m, void *v) · line 3608
Quoted 4 lines
mm/memcontrol.c +4
-0+rcu_read_lock (); + memcg =mem_cgroup_from_task (p); +if (idx !=-1 ) +this_cpu_inc (memcg->stat_numa->locality[idx]);I thought cgroups were supposed to be hierarchical. That is, if we have:
R / \ A /\ B \ t1
Quoted 2 lines
Then our task t1 should be accounted to B (as you do), but also to A and R.
I get the point but not quite sure about this...
Not like pages there are no hierarchical limitation on locality, also tasks running in a particular group have no influence to others, not to mention the extra overhead, does it really meaningful to account the stuff hierarchically?
Regards,
Michael Wang
Quoted 4 lines
Quoted 4 lines
mm/memcontrol.c +3
-0+ rcu_read_unlock(); +} +#endif
Peter Zijlstra
maintainerRe: [PATCH 1/4] numa: introduce per-cgroup numa balancing locality, statistic
Quoted 19 lines · 王贇
On 2019/7/11 下午9:47, Peter Zijlstra wrote:
[snip]Quoted 15 lines
mm/memcontrol.c static int memcg_numa_stat_show(struct seq_file *m, void *v) · line 3608
Quoted 4 lines
mm/memcontrol.c +4
-0+rcu_read_lock (); + memcg =mem_cgroup_from_task (p); +if (idx !=-1 ) +this_cpu_inc (memcg->stat_numa->locality[idx]);I thought cgroups were supposed to be hierarchical. That is, if we have:
R / \ A /\ B \ t1
Quoted 6 lines
Quoted 2 lines
Then our task t1 should be accounted to B (as you do), but also to A and R.
I get the point but not quite sure about this...
Not like pages there are no hierarchical limitation on locality, also tasks
You can use cpusets to affect that.
Quoted 2 lines
running in a particular group have no influence to others, not to mention the extra overhead, does it really meaningful to account the stuff hierarchically?
AFAIU it's a requirement of cgroups to be hierarchical. All our other cgroup accounting is like that.
Re: [PATCH 1/4] numa: introduce per-cgroup numa balancing locality, statistic
On 2019/7/12 下午3:58, Peter Zijlstra wrote:
[snip]
Quoted 9 lines
Quoted 7 lines
Quoted 3 lines
Then our task t1 should be accounted to B (as you do), but also to A and R.
I get the point but not quite sure about this...
Not like pages there are no hierarchical limitation on locality, also tasks
You can use cpusets to affect that.
Could you please give more detail on this?
Quoted 6 lines
Quoted 3 lines
running in a particular group have no influence to others, not to mention the extra overhead, does it really meaningful to account the stuff hierarchically?
AFAIU it's a requirement of cgroups to be hierarchical. All our other cgroup accounting is like that.
Ok, should respect the convention :-)
Regards,
Michael Wang
Quoted 1 line
Peter Zijlstra
maintainerRe: [PATCH 1/4] numa: introduce per-cgroup numa balancing locality, statistic
Quoted 8 lines · 王贇
On 2019/7/12 下午3:58, Peter Zijlstra wrote:
[snip]Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Then our task t1 should be accounted to B (as you do), but also to A and R.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I get the point but not quite sure about this...
Not like pages there are no hierarchical limitation on locality, also tasks
You can use cpusets to affect that.
Could you please give more detail on this?
Documentation/cgroup-v1/cpusets.txt
Look for mems_allowed.
Re: [PATCH 1/4] numa: introduce per-cgroup numa balancing locality, statistic
Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · 王贇
On 2019/7/12 下午3:58, Peter Zijlstra wrote:
[snip]Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Then our task t1 should be accounted to B (as you do), but also to A and R.
I get the point but not quite sure about this...
Not like pages there are no hierarchical limitation on locality, also tasks
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
You can use cpusets to affect that.
Could you please give more detail on this?
Documentation/cgroup-v1/cpusets.txtLook for
mems_allowed.
This is the attribute belong to cpuset cgroup isn't it?
Forgive me but I have no idea on how to combined this with memory cgroup's locality hierarchical update... parent memory cgroup do not have influence on mems_allowed to it's children, correct?
What about we just account the locality status of child memory group into it's ancestors?
Regards,
Michael Wang
Quoted 1 line
Re: [PATCH 1/4] numa: introduce per-cgroup numa balancing locality, statistic
On 2019/7/12 下午6:10, 王贇 wrote:
[snip]
Quoted 7 lines
Quoted 4 lines
Documentation/cgroup-v1/cpusets.txtLook for
mems_allowed.This is the attribute belong to cpuset cgroup isn't it?
Quoted 7 lines
Forgive me but I have no idea on how to combined this with memory cgroup's locality hierarchical update... parent memory cgroup do not have influence on
mems_allowedto it's children, correct?What about we just account the locality status of child memory group into it's ancestors?
We have rethink about this, and found no strong reason to stay with memory cgroup anymore.
We used to acquire pages number, exectime and locality together from memory cgroup, to make thing easier for our numa balancer module, as now we use the numa group approach, maybe we can just move these accounting into cpu cgroups, so all these features stay in one subsys and could be hierarchical :-)
Regards,
Michael Wang
Quoted 5 lines
Regards,
Michael WangQuoted 1 line
Michal Koutný
maintainerRe: [PATCH 1/4] numa: introduce per-cgroup numa balancing locality, statistic
In reply to 王贇Hello Yun.
Quoted 4 lines · 王贇
Forgive me but I have no idea on how to combined this with memory cgroup's locality hierarchical update... parent memory cgroup do not have influence on
mems_allowedto it's children, correct?
I'd recommend to look at the v2 of the cpuset controller that implements the hierarchical behavior among configured memory node sets.
(My comment would better fit to
[PATCH 3/4] numa: introduce numa group per task group
IIUC, you could use cpuset controller to constraint memory nodes.)
For the second part (accessing numa statistics, i.e. this patch), I wonder wheter this information wouldn't be better presented under the cpuset controller too.
HTH,
Michal
Re: [PATCH 1/4] numa: introduce per-cgroup numa balancing locality, statistic
Hi Michal,
Thx for the comments :-)
Quoted 9 lines · Michal Koutný
Hello Yun.
Quoted 4 lines · 王贇
Forgive me but I have no idea on how to combined this with memory cgroup's locality hierarchical update... parent memory cgroup do not have influence on
mems_allowedto it's children, correct?I'd recommend to look at the v2 of the cpuset controller that implements the hierarchical behavior among configured memory node sets.
Actually whatever the memory node sets or cpu allow sets is, it will take effect on task's behavior regarding memory location and cpu location, while the locality only care about the results rather than the sets.
For example if we bind tasks to cpus of node 0 and memory allow only the node 1, by cgroup controller or madvise, then they will running on node 0 with all the memory on node 1, on each PF for numa balancing, the task will access page on node 1 from node 0 remotely, so the locality will always be 0.
Quoted 8 lines
(My comment would better fit to
[PATCH 3/4] numa: introduce numa group per task group
IIUC, you could use cpuset controller to constraint memory nodes.)For the second part (accessing numa statistics, i.e. this patch), I wonder wheter this information wouldn't be better presented under the cpuset controller too.
Yeah, we realized the cpu cgroup could be a better place to hold these new statistics, both locality and exectime are task's running behavior, related to memory location but not the memory behavior, will apply in next version.
Regards,
Michael Wang
Quoted 4 lines
HTH,
Michal
Michal Koutný
maintainerRe: [PATCH 1/4] numa: introduce per-cgroup numa balancing locality, statistic
Quoted 4 lines · 王贇
Actually whatever the memory node sets or cpu allow sets is, it will take effect on task's behavior regarding memory location and cpu location, while the locality only care about the results rather than the sets.
My previous response missed much of the context, so it was a bit off.
I see what you mean by the locality now. Alas, I can't assess whether it's the right thing to do regarding NUMA behavior that you try to optimize (i.e. you need an answer from someone more familiar with NUMA balancing).
Michal
PATCH 2/4 numa: append per-node execution info in memory.numa_stat
In reply to 王贇This patch introduced numa execution information, to imply the numa efficiency.
By doing 'cat /sys/fs/cgroup/memory/CGROUP_PATH/memory.numa_stat', we see new output line heading with 'exectime', like:
exectime 311900 407166
which means the tasks of this cgroup executed 311900 micro seconds on node 0, and 407166 ms on node 1.
Combined with the memory node info, we can estimate the numa efficiency, for example if the node memory info is:
total=206892 N0=21933 N1=185171
By monitoring the increments, if the topology keep in this way and locality is not nice, then it imply numa balancing can't help migrate the memory from node 1 to 0 which is accessing by tasks on node 0, or tasks can't migrate to node 1 for some reason, then you may consider to bind the cgroup on the cpus of node 1.
--- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 13 +++++++++++++ 2 files changed, 14 insertions(+)
include/linux/memcontrol.h +1 -0
@@ -190,6 +190,7 @@ enum memcg_numa_locality_interval {struct memcg_stat_numa {u64 locality[NR_NL_INTERVAL ]; +u64 exectime; };#endif
mm/memcontrol.c +13 -0
@@ -3575,6 +3575,18 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) seq_printf(m, " %u", jiffies_to_msecs(sum)); }seq_putc (m,'\n' ); + +seq_puts (m,"exectime" ); +for_each_online_node (nr) { +int cpu; +u64 sum =0 ; + +for_each_cpu (cpu,cpumask_of_node (nr)) + sum +=per_cpu (memcg->stat_numa->exectime, cpu); + +seq_printf (m," %llu" ,jiffies_to_msecs (sum)); + } +seq_putc (m,'\n' );#endif return 0 ;
@@ -3606,6 +3618,7 @@ void memcg_stat_numa_update(struct task_struct *p) memcg =mem_cgroup_from_task (p);if (idx !=-1 )this_cpu_inc (memcg->stat_numa->locality[idx]); +this_cpu_inc (memcg->stat_numa->exectime); rcu_read_unlock(); }#endif
Signature · 1 line
2.14.4.44.g2045bb6
Peter Zijlstra
maintainerRe: [PATCH 2/4] numa: append per-node execution info in memory.numa_stat
include/linux/memcontrol.h enum memcg_numa_locality_interval { · line 193
Quoted 6 lines · 王贇
+++
b/include/linux/memcontrol.hinclude/linux/memcontrol.h +1
-0@@ -190,6 +190,7 @@ enum memcg_numa_locality_interval {struct memcg_stat_numa {u64 locality[NR_NL_INTERVAL ]; +u64 exectime;
Maybe call the field jiffies, because that's what it counts.
Quoted 26 lines
(quoted diff) +0
-0}; #endifmm/memcontrol.c +12
-0@@ -3575,6 +3575,18 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) seq_printf(m, " %u", jiffies_to_msecs(sum)); }seq_putc (m,'\n' ); + +seq_puts (m,"exectime" ); +for_each_online_node (nr) { +int cpu; +u64 sum =0 ; + +for_each_cpu (cpu,cpumask_of_node (nr)) + sum +=per_cpu (memcg->stat_numa->exectime, cpu); + +seq_printf (m," %llu" ,jiffies_to_msecs (sum)); + } +seq_putc (m,'\n' );#endif return 0 ;
Re: [PATCH 2/4] numa: append per-node execution info in memory.numa_stat
Quoted 10 lines · Peter Zijlstra
include/linux/memcontrol.h enum memcg_numa_locality_interval { · line 193
Quoted 6 lines · 王贇
+++
b/include/linux/memcontrol.hinclude/linux/memcontrol.h +1
-0@@ -190,6 +190,7 @@ enum memcg_numa_locality_interval {struct memcg_stat_numa {u64 locality[NR_NL_INTERVAL ]; +u64 exectime;Maybe call the field jiffies, because that's what it counts.
Sure, will be in next version.
Regards,
Michael Wang
Quoted 27 lines
Quoted 27 lines
(quoted diff) +0
-0}; #endifmm/memcontrol.c +12
-0@@ -3575,6 +3575,18 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) seq_printf(m, " %u", jiffies_to_msecs(sum)); }seq_putc (m,'\n' ); + +seq_puts (m,"exectime" ); +for_each_online_node (nr) { +int cpu; +u64 sum =0 ; + +for_each_cpu (cpu,cpumask_of_node (nr)) + sum +=per_cpu (memcg->stat_numa->exectime, cpu); + +seq_printf (m," %llu" ,jiffies_to_msecs (sum)); + } +seq_putc (m,'\n' );#endif return 0 ;
PATCH 3/4 numa: introduce numa group per task group
In reply to 王贇By tracing numa page faults, we recognize tasks sharing the same page, and try pack them together into a single numa group.
However when two task share lot's of cache pages while not much anonymous pages, since numa balancing do not tracing cache page, they have no chance to join into the same group.
While tracing cache page cost too much, we could use some hints from userland and cpu cgroup could be a good one.
This patch introduced new entry 'numa_group' for cpu cgroup, by echo non-zero into the entry, we can now force all the tasks of this cgroup to join the same numa group serving for task group.
In this way tasks are more likely to settle down on the same node, to share closer cpu cache and gain benefit from NUMA on both file/anonymous pages.
Besides, when multiple cgroup enabled numa group, they will be able to exchange task location by utilizing numa migration, in this way they could achieve single node settle down without breaking load balance.
--- kernel/sched/core.c | 37 +++++++++++ kernel/sched/fair.c | 175 ++++++++++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 14 +++++ 3 files changed, 225 insertions(+), 1 deletion(-)
kernel/sched/core.c +37 -0
@@ -6790,6 +6790,8 @@ void sched_offline_group(struct task_group *tg) {unsigned long flags; +update_tg_numa_group (tg, false); +/* End participation in shares distribution: */ unregister_fair_sched_group (tg);
@@ -7277,6 +7279,34 @@ static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, }#endif /* CONFIG_RT_GROUP_SCHED */ +#ifdef CONFIG_NUMA_BALANCING +static DEFINE_MUTEX (numa_mutex); + +static int cpu_numa_group_show (struct seq_file *sf,void *v) +{ +struct task_group *tg =css_tg (seq_css (sf)); + +mutex_lock (&numa_mutex); +show_tg_numa_group (tg, sf); +mutex_unlock (&numa_mutex); + +return 0 ; +} + +static int cpu_numa_group_write_s64 (struct cgroup_subsys_state *css, +struct cftype *cft,s64 numa_group) +{ +int ret; +struct task_group *tg =css_tg (css); + +mutex_lock (&numa_mutex); + ret =update_tg_numa_group (tg, numa_group); +mutex_unlock (&numa_mutex); + +return ret; +} +#endif /* CONFIG_NUMA_BALANCING */ + static struct cftype cpu_legacy_files[] = {#ifdef CONFIG_FAIR_GROUP_SCHED {
@@ -7312,6 +7342,13 @@ static struct cftype cpu_legacy_files[] = { .read_u64 = cpu_rt_period_read_uint, .write_u64 = cpu_rt_period_write_uint, }, +#endif +#ifdef CONFIG_NUMA_BALANCING + { + .name = "numa_group", + .write_s64 = cpu_numa_group_write_s64, + .seq_show = cpu_numa_group_show, + }, #endif { }/* Terminate */ };
kernel/sched/fair.c +174 -1
193 lines not on this page · raw message
kernel/sched/sched.h +14 -0
@@ -393,6 +393,10 @@ struct task_group {#endif struct cfs_bandwidth cfs_bandwidth; + +#ifdef CONFIG_NUMA_BALANCING +void *numa_group; +#endif };#ifdef CONFIG_FAIR_GROUP_SCHED
@@ -1286,11 +1290,21 @@ extern int migrate_task_to(struct task_struct *p, int cpu);extern int migrate_swap (struct task_struct *p,struct task_struct *t,int cpu,int scpu);extern void init_numa_balancing (unsigned long clone_flags,struct task_struct *p); +extern void show_tg_numa_group (struct task_group *tg,struct seq_file *sf); +extern int update_tg_numa_group (struct task_group *tg,bool numa_group);#else static inline void init_numa_balancing (unsigned long clone_flags,struct task_struct *p) { } +static inline void +show_tg_numa_group (struct task_group *tg,struct seq_file *sf) +{ +} +update_tg_numa_group(struct task_group *tg, bool numa_group) +{ +return 0 ; +}#endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_SMP
Signature · 1 line
2.14.4.44.g2045bb6
Peter Zijlstra
maintainerRe: [PATCH 3/4] numa: introduce numa group per task group
Quoted 8 lines · 王贇
By tracing numa page faults, we recognize tasks sharing the same page, and try pack them together into a single numa group.
However when two task share lot's of cache pages while not much anonymous pages, since numa balancing do not tracing cache page, they have no chance to join into the same group.
While tracing cache page cost too much, we could use some hints from
I forgot; where again do we skip shared pages? task_numa_work() doesn't seem to skip file vmas.
Quoted 6 lines
userland and cpu cgroup could be a good one.
This patch introduced new entry '
numa_group' for cpu cgroup, by echo non-zero into the entry, we can now force all the tasks of this cgroup to join the same numa group serving for task group.
Quoted 7 lines
In this way tasks are more likely to settle down on the same node, to share closer cpu cache and gain benefit from NUMA on both file/anonymous pages.
Besides, when multiple cgroup enabled numa group, they will be able to exchange task location by utilizing numa migration, in this way they could achieve single node settle down without breaking load balance.
I dislike cgroup only interfaces; it there really nothing else we could use for this?
Re: [PATCH 3/4] numa: introduce numa group per task group
Quoted 4 lines · Peter Zijlstra
Quoted 3 lines · 王贇
By tracing numa page faults, we recognize tasks sharing the same page, and try pack them together into a single numa group.
Quoted 8 lines
Quoted 5 lines
However when two task share lot's of cache pages while not much anonymous pages, since numa balancing do not tracing cache page, they have no chance to join into the same group.
While tracing cache page cost too much, we could use some hints from
I forgot; where again do we skip shared pages?
task_numa_work()doesn't seem to skip file vmas.
That's the page cache generated by file read/write, rather than the pages for file mapping, pages of memory to support IO also won't be considered as shared between tasks since they don't belong to any particular task, but may serving multiples.
Quoted 11 lines
Quoted 11 lines
userland and cpu cgroup could be a good one.
This patch introduced new entry '
numa_group' for cpu cgroup, by echo non-zero into the entry, we can now force all the tasks of this cgroup to join the same numa group serving for task group.In this way tasks are more likely to settle down on the same node, to share closer cpu cache and gain benefit from NUMA on both file/anonymous pages.
Quoted 6 lines
Quoted 3 lines
Besides, when multiple cgroup enabled numa group, they will be able to exchange task location by utilizing numa migration, in this way they could achieve single node settle down without breaking load balance.
I dislike cgroup only interfaces; it there really nothing else we could use for this?
Me too... while at this moment that's the best approach we have got, we also tried to use separately module to handle these automatically, but this need a very good understanding of the system, configuration and workloads which only known by the owner.
So maybe just providing the functionality and leave the choice to user is not that bad?
Regards,
Michael Wang
Quoted 1 line
PATCH 4/4 numa: introduce numa cling feature
In reply to 王贇Although we paid so many effort to settle down task on a particular node, there are still chances for a task to leave it's preferred node, that is by wakeup, numa swap migrations or load balance.
When we are using cpu cgroup in share way, since all the workloads see all the cpus, it could be really bad especially when there are too many fast wakeup, although now we can numa group the tasks, they won't really stay on the same node, for example we have numa group ng_A, ng_B, ng_C, ng_D, it's very likely result as:
CPU Usage: Node 0 Node 1 ng_A(600%) ng_A(400%) ng_B(400%) ng_B(600%) ng_C(400%) ng_C(600%) ng_D(600%) ng_D(400%) Memory Ratio: Node 0 Node 1 ng_A(60%) ng_A(40%) ng_B(40%) ng_B(60%) ng_C(40%) ng_C(60%) ng_D(60%) ng_D(40%)
Locality won't be too bad but far from the best situation, we want a numa group to settle down thoroughly on a particular node, with every thing balanced.
Thus we introduce the numa cling, which try to prevent tasks leaving the preferred node on wakeup fast path.
This help thoroughly settle down the workloads on single node, but when multiple numa group try to settle down on the same node, unbalancing could happen.
For example we have numa group ng_A, ng_B, ng_C, ng_D, it may result in situation like:
CPU Usage:
Node 0 Node 1 ng_A(1000%) ng_B(1000%) ng_C(400%) ng_C(600%) ng_D(400%) ng_D(600%)
Memory Ratio:
Node 0 Node 1 ng_A(100%) ng_B(100%) ng_C(10%) ng_C(90%) ng_D(10%) ng_D(90%)
This is because when ng_C, ng_D start to have most of the memory on node 1 at some point, task_x of ng_C stay on node 0 will try to do numa swap migration with the task_y of ng_D stay on node 1 as long as load balanced, the result is task_x stay on node 1 and task_y stay on node 0, while both of them prefer node 1.
Now when other tasks of ng_D stay on node 1 wakeup task_y, task_y will very likely go back to node 1, and since numa cling enabled, it will keep stay on node 1 although load unbalanced, this could be frequently and more and more tasks will prefer the node 1 and make it busy.
So the key point here is to stop doing numa cling when load starting to become unbalancing.
We achieved this by monitoring the migration failure ratio, in scenery above, too much tasks prefer node 1 and will keep migrating to it, load unbalancing could lead into the migration failure in this case, and when the failure ratio above the specified degree, we pause the cling and try to resettle the workloads on a better node by stop tasks prefer the busy node, this will finally give us the result like:
CPU Usage:
Node 0 Node 1 ng_A(1000%) ng_B(1000%) ng_C(1000%) ng_D(1000%)
Memory Ratio:
Node 0 Node 1 ng_A(100%) ng_B(100%) ng_C(100%) ng_D(100%)
Now we achieved the best locality and maximum hot cache benefit.
Tested on a 2 node box with 96 cpus, do sysbench-mysql-oltp_read_write testing, X mysqld instances created and attached to X cgroups, X sysbench instances then created and attached to corresponding cgroup to test the mysql with oltp_read_write script for 20 minutes, average eps show:
origin ng + cling
4 instances each 24 threads 7545.28 7790.49 +3.25%
4 instances each 48 threads 9359.36 9832.30 +5.05%
4 instances each 72 threads 9602.88 10196.95 +6.19%
8 instances each 24 threads 4478.82 4508.82 +0.67%
8 instances each 48 threads 5514.90 5689.93 +3.17%
8 instances each 72 threads 5582.19 5741.33 +2.85%
Also tested with perf-bench-numa, dbench, sysbench-memory, pgbench, tiny improvement observed.
--- include/linux/sched/sysctl.h | 3 + kernel/sched/fair.c | 283 +++++++++++++++++++++++++++++++++++++++++-- kernel/sysctl.c | 9 ++ 3 files changed, 283 insertions(+), 12 deletions(-)
include/linux/sched/sysctl.h +3 -0
@@ -38,6 +38,9 @@ extern unsigned int sysctl_numa_balancing_scan_period_min;extern unsigned int sysctl_numa_balancing_scan_period_max;extern unsigned int sysctl_numa_balancing_scan_size; +extern unsigned int sysctl_numa_balancing_cling_degree; +extern unsigned int max_numa_balancing_cling_degree; +#ifdef CONFIG_SCHED_DEBUG extern __read_mostly unsigned int sysctl_sched_migration_cost;extern __read_mostly unsigned int sysctl_sched_nr_migrate;
kernel/sched/fair.c +271 -12
395 lines not on this page · raw message
kernel/sysctl.c +9 -0
@@ -417,6 +417,15 @@ static struct ctl_table kern_table[] = { .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ONE, }, + { + .procname = "numa_balancing_cling_degree", + .data = &sysctl_numa_balancing_cling_degree, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = &max_numa_balancing_cling_degree, + }, { .procname = "numa_balancing", .data = NULL, /* filled in by handler */
Signature · 1 line
2.14.4.44.g2045bb6
PATCH v2 4/4 numa: introduce numa cling feature
Although we paid so many effort to settle down task on a particular node, there are still chances for a task to leave it's preferred node, that is by wakeup, numa swap migrations or load balance.
When we are using cpu cgroup in share way, since all the workloads see all the cpus, it could be really bad especially when there are too many fast wakeup, although now we can numa group the tasks, they won't really stay on the same node, for example we have numa group ng_A, ng_B, ng_C, ng_D, it's very likely result as:
CPU Usage: Node 0 Node 1 ng_A(600%) ng_A(400%) ng_B(400%) ng_B(600%) ng_C(400%) ng_C(600%) ng_D(600%) ng_D(400%) Memory Ratio: Node 0 Node 1 ng_A(60%) ng_A(40%) ng_B(40%) ng_B(60%) ng_C(40%) ng_C(60%) ng_D(60%) ng_D(40%)
Locality won't be too bad but far from the best situation, we want a numa group to settle down thoroughly on a particular node, with every thing balanced.
Thus we introduce the numa cling, which try to prevent tasks leaving the preferred node on wakeup fast path.
This help thoroughly settle down the workloads on single node, but when multiple numa group try to settle down on the same node, unbalancing could happen.
For example we have numa group ng_A, ng_B, ng_C, ng_D, it may result in situation like:
CPU Usage:
Node 0 Node 1 ng_A(1000%) ng_B(1000%) ng_C(400%) ng_C(600%) ng_D(400%) ng_D(600%)
Memory Ratio:
Node 0 Node 1 ng_A(100%) ng_B(100%) ng_C(10%) ng_C(90%) ng_D(10%) ng_D(90%)
This is because when ng_C, ng_D start to have most of the memory on node 1 at some point, task_x of ng_C stay on node 0 will try to do numa swap migration with the task_y of ng_D stay on node 1 as long as load balanced, the result is task_x stay on node 1 and task_y stay on node 0, while both of them prefer node 1.
Now when other tasks of ng_D stay on node 1 wakeup task_y, task_y will very likely go back to node 1, and since numa cling enabled, it will keep stay on node 1 although load unbalanced, this could be frequently and more and more tasks will prefer the node 1 and make it busy.
So the key point here is to stop doing numa cling when load starting to become unbalancing.
We achieved this by monitoring the migration failure ratio, in scenery above, too much tasks prefer node 1 and will keep migrating to it, load unbalancing could lead into the migration failure in this case, and when the failure ratio above the specified degree, we pause the cling and try to resettle the workloads on a better node by stop tasks prefer the busy node, this will finally give us the result like:
CPU Usage:
Node 0 Node 1 ng_A(1000%) ng_B(1000%) ng_C(1000%) ng_D(1000%)
Memory Ratio:
Node 0 Node 1 ng_A(100%) ng_B(100%) ng_C(100%) ng_D(100%)
Now we achieved the best locality and maximum hot cache benefit.
Tested on a 2 node box with 96 cpus, do sysbench-mysql-oltp_read_write testing, X mysqld instances created and attached to X cgroups, X sysbench instances then created and attached to corresponding cgroup to test the mysql with oltp_read_write script for 20 minutes, average eps show:
origin ng + cling
4 instances each 24 threads 7545.28 7790.49 +3.25%
4 instances each 48 threads 9359.36 9832.30 +5.05%
4 instances each 72 threads 9602.88 10196.95 +6.19%
8 instances each 24 threads 4478.82 4508.82 +0.67%
8 instances each 48 threads 5514.90 5689.93 +3.17%
8 instances each 72 threads 5582.19 5741.33 +2.85%
Also tested with perf-bench-numa, dbench, sysbench-memory, pgbench, tiny improvement observed.
---
v2:
* migrate_degrades_locality() now return 1 when numa cling to source node
include/linux/sched/sysctl.h | 3 + kernel/sched/fair.c | 287 +++++++++++++++++++++++++++++++++++++++++-- kernel/sysctl.c | 9 ++ 3 files changed, 286 insertions(+), 13 deletions(-)
include/linux/sched/sysctl.h +3 -0
@@ -38,6 +38,9 @@ extern unsigned int sysctl_numa_balancing_scan_period_min;extern unsigned int sysctl_numa_balancing_scan_period_max;extern unsigned int sysctl_numa_balancing_scan_size; +extern unsigned int sysctl_numa_balancing_cling_degree; +extern unsigned int max_numa_balancing_cling_degree; +#ifdef CONFIG_SCHED_DEBUG extern __read_mostly unsigned int sysctl_sched_migration_cost;extern __read_mostly unsigned int sysctl_sched_nr_migrate;
kernel/sched/fair.c +274 -13
411 lines not on this page · raw message
kernel/sysctl.c +9 -0
@@ -417,6 +417,15 @@ static struct ctl_table kern_table[] = { .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ONE, }, + { + .procname = "numa_balancing_cling_degree", + .data = &sysctl_numa_balancing_cling_degree, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = &max_numa_balancing_cling_degree, + }, { .procname = "numa_balancing", .data = NULL, /* filled in by handler */
Signature · 1 line
2.14.4.44.g2045bb6
Re: [PATCH v2 4/4] numa: introduce numa cling feature
Quoted 8 lines · Hillf Danton
Quoted 6 lines · On Mon, 8 Jul 2019 10:25:27 +0800 Michael Wang wrote:
/* Attempt to migrate a task to a CPU on the preferred node. */ static void
numa_migrate_preferred(struct task_struct *p){kernel/sched/fair.c +1
-0+bool failed, target;unsigned long interval = HZ;
Quoted 8 lines
kernel/sched/fair.c static void numa_migrate_preferred(struct task_struct *p) · line 2121
Quoted 7 lines
/* This task has no NUMA fault statistics yet */kernel/sched/fair.c +2
-0@@ -1891,8 +2117,12 @@ static void numa_migrate_preferred(struct task_struct *p)if (task_node (p) == p->numa_preferred_nid)return ; + target = p->numa_preferred_nid; +Something instead of bool can be used, too.
Thx for point out :-) to be fix in v3.
Quoted 13 lines
Quoted 13 lines
kernel/sched/fair.c +3
-1/* Otherwise, try migrate to a CPU on the preferred node */ -task_numa_migrate (p); + failed = (task_numa_migrate (p) !=0 ); + +update_migrate_stat (p, target, failed); }static void
kernel/sched/fair.c +0
-0@@ -6195,6 +6447,13 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)if ((unsigned )i < nr_cpumask_bits)return i;
Quoted 8 lines
kernel/sched/fair.c static int select_idle_sibling(struct task_struct *p, int prev, int target) · line 6456
Quoted 7 lines
kernel/sched/fair.c +7
-0+/* +* Failed to find an idle cpu, wake affine may want to pull but +* try stay on prev-cpu when the task cling to it. +*/ +if (task_numa_cling (p,cpu_to_node (prev),cpu_to_node (target))) +return prev; +Curious to know what test figures would look like without the above line.
It depends on the wake affine condition then, when waker task consider wakee suitable for pull, wakee may leave the preferred node, or maybe pull to the preferred node, just randomly and follow the fate.
In mysql case when there are many such wakeup cases and system is very busy, the observed workloads could be 4:6 or 3:7 distributed in two nodes.
Regards,
Michael Wang
Quoted 18 lines
Quoted 17 lines
kernel/sched/fair.c +0
-0return target; }Tested on a 2 node box with 96 cpus, do sysbench-mysql-oltp_read_write testing, X mysqld instances created and attached to X cgroups, X sysbench instances then created and attached to corresponding cgroup to test the mysql with
oltp_read_writescript for 20 minutes, average eps show:origin ng + cling
4 instances each 24 threads 7545.28 7790.49 +3.25%
4 instances each 48 threads 9359.36 9832.30 +5.05%
4 instances each 72 threads 9602.88 10196.95 +6.19%8 instances each 24 threads 4478.82 4508.82 +0.67%
8 instances each 48 threads 5514.90 5689.93 +3.17%
8 instances each 72 threads 5582.19 5741.33 +2.85%
PATCH v3 4/4 numa: introduce numa cling feature
In reply to 王贇Although we paid so many effort to settle down task on a particular node, there are still chances for a task to leave it's preferred node, that is by wakeup, numa swap migrations or load balance.
When we are using cpu cgroup in share way, since all the workloads see all the cpus, it could be really bad especially when there are too many fast wakeup, although now we can numa group the tasks, they won't really stay on the same node, for example we have numa group ng_A, ng_B, ng_C, ng_D, it's very likely result as:
CPU Usage: Node 0 Node 1 ng_A(600%) ng_A(400%) ng_B(400%) ng_B(600%) ng_C(400%) ng_C(600%) ng_D(600%) ng_D(400%) Memory Ratio: Node 0 Node 1 ng_A(60%) ng_A(40%) ng_B(40%) ng_B(60%) ng_C(40%) ng_C(60%) ng_D(60%) ng_D(40%)
Locality won't be too bad but far from the best situation, we want a numa group to settle down thoroughly on a particular node, with every thing balanced.
Thus we introduce the numa cling, which try to prevent tasks leaving the preferred node on wakeup fast path.
This help thoroughly settle down the workloads on single node, but when multiple numa group try to settle down on the same node, unbalancing could happen.
For example we have numa group ng_A, ng_B, ng_C, ng_D, it may result in situation like:
CPU Usage:
Node 0 Node 1 ng_A(1000%) ng_B(1000%) ng_C(400%) ng_C(600%) ng_D(400%) ng_D(600%)
Memory Ratio:
Node 0 Node 1 ng_A(100%) ng_B(100%) ng_C(10%) ng_C(90%) ng_D(10%) ng_D(90%)
This is because when ng_C, ng_D start to have most of the memory on node 1 at some point, task_x of ng_C stay on node 0 will try to do numa swap migration with the task_y of ng_D stay on node 1 as long as load balanced, the result is task_x stay on node 1 and task_y stay on node 0, while both of them prefer node 1.
Now when other tasks of ng_D stay on node 1 wakeup task_y, task_y will very likely go back to node 1, and since numa cling enabled, it will keep stay on node 1 although load unbalanced, this could be frequently and more and more tasks will prefer the node 1 and make it busy.
So the key point here is to stop doing numa cling when load starting to become unbalancing.
We achieved this by monitoring the migration failure ratio, in scenery above, too much tasks prefer node 1 and will keep migrating to it, load unbalancing could lead into the migration failure in this case, and when the failure ratio above the specified degree, we pause the cling and try to resettle the workloads on a better node by stop tasks prefer the busy node, this will finally give us the result like:
CPU Usage:
Node 0 Node 1 ng_A(1000%) ng_B(1000%) ng_C(1000%) ng_D(1000%)
Memory Ratio:
Node 0 Node 1 ng_A(100%) ng_B(100%) ng_C(100%) ng_D(100%)
Now we achieved the best locality and maximum hot cache benefit.
Tested on a 2 node box with 96 cpus, do sysbench-mysql-oltp_read_write testing, X mysqld instances created and attached to X cgroups, X sysbench instances then created and attached to corresponding cgroup to test the mysql with oltp_read_write script for 20 minutes, average eps show:
origin ng + cling
4 instances each 24 threads 7545.28 7790.49 +3.25%
4 instances each 48 threads 9359.36 9832.30 +5.05%
4 instances each 72 threads 9602.88 10196.95 +6.19%
8 instances each 24 threads 4478.82 4508.82 +0.67%
8 instances each 48 threads 5514.90 5689.93 +3.17%
8 instances each 72 threads 5582.19 5741.33 +2.85%
Also tested with perf-bench-numa, dbench, sysbench-memory, pgbench, tiny improvement observed.
---
v3:
* fix target type in numa_migrate_preferred()
include/linux/sched/sysctl.h | 3 + kernel/sched/fair.c | 288 +++++++++++++++++++++++++++++++++++++++++-- kernel/sysctl.c | 9 ++ 3 files changed, 287 insertions(+), 13 deletions(-)
include/linux/sched/sysctl.h +3 -0
@@ -38,6 +38,9 @@ extern unsigned int sysctl_numa_balancing_scan_period_min;extern unsigned int sysctl_numa_balancing_scan_period_max;extern unsigned int sysctl_numa_balancing_scan_size; +extern unsigned int sysctl_numa_balancing_cling_degree; +extern unsigned int max_numa_balancing_cling_degree; +#ifdef CONFIG_SCHED_DEBUG extern __read_mostly unsigned int sysctl_sched_migration_cost;extern __read_mostly unsigned int sysctl_sched_nr_migrate;
kernel/sched/fair.c +275 -13
412 lines not on this page · raw message
kernel/sysctl.c +9 -0
@@ -417,6 +417,15 @@ static struct ctl_table kern_table[] = { .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ONE, }, + { + .procname = "numa_balancing_cling_degree", + .data = &sysctl_numa_balancing_cling_degree, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = &max_numa_balancing_cling_degree, + }, { .procname = "numa_balancing", .data = NULL, /* filled in by handler */
Signature · 1 line
2.14.4.44.g2045bb6
Peter Zijlstra
maintainerRe: [PATCH 4/4] numa: introduce numa cling feature
In reply to 王贇Quoted 35 lines · 王贇
Although we paid so many effort to settle down task on a particular node, there are still chances for a task to leave it's preferred node, that is by wakeup, numa swap migrations or load balance.
When we are using cpu cgroup in share way, since all the workloads see all the cpus, it could be really bad especially when there are too many fast wakeup, although now we can numa group the tasks, they won't really stay on the same node, for example we have numa group
ng_A,ng_B,ng_C,ng_D, it's very likely result as:CPU Usage: Node 0 Node 1 ng_A(600%) ng_A(400%) ng_B(400%) ng_B(600%) ng_C(400%) ng_C(600%) ng_D(600%) ng_D(400%) Memory Ratio: Node 0 Node 1 ng_A(60%) ng_A(40%) ng_B(40%) ng_B(60%) ng_C(40%) ng_C(60%) ng_D(60%) ng_D(40%)Locality won't be too bad but far from the best situation, we want a numa group to settle down thoroughly on a particular node, with every thing balanced.
Thus we introduce the numa cling, which try to prevent tasks leaving the preferred node on wakeup fast path.
kernel/sched/fair.c +0
-0@@ -6195,6 +6447,13 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)if ((unsigned )i < nr_cpumask_bits)return i;
kernel/sched/fair.c static int select_idle_sibling(struct task_struct *p, int prev, int target) · line 6458
Quoted 10 lines
kernel/sched/fair.c +7
-0+/* +* Failed to find an idle cpu, wake affine may want to pull but +* try stay on prev-cpu when the task cling to it. +*/ +if (task_numa_cling (p,cpu_to_node (prev),cpu_to_node (target))) +return prev; +return target; }
Select idle sibling should never cross node boundaries and is thus the entirely wrong place to fix anything.
Re: [PATCH 4/4] numa: introduce numa cling feature
On 2019/7/11 下午10:27, Peter Zijlstra wrote:
[snip]
Quoted 8 lines
Quoted 8 lines
Thus we introduce the numa cling, which try to prevent tasks leaving the preferred node on wakeup fast path.
kernel/sched/fair.c +0
-0@@ -6195,6 +6447,13 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)if ((unsigned )i < nr_cpumask_bits)return i;
Quoted 12 lines
kernel/sched/fair.c static int select_idle_sibling(struct task_struct *p, int prev, int target) · line 6458
Quoted 9 lines
kernel/sched/fair.c +7
-0+/* +* Failed to find an idle cpu, wake affine may want to pull but +* try stay on prev-cpu when the task cling to it. +*/ +if (task_numa_cling (p,cpu_to_node (prev),cpu_to_node (target))) +return prev; +return target; }Select idle sibling should never cross node boundaries and is thus the entirely wrong place to fix anything.
Hmm.. in our early testing the printk show both select_task_rq_fair() and task_numa_find_cpu() will call select_idle_sibling with prev and target on different node, thus we pick this point to save few lines.
But if the semantics of select_idle_sibling() is to return cpu on the same node of target, what about move the logical after select_idle_sibling() for the two callers?
Regards,
Michael Wang
Quoted 1 line
Peter Zijlstra
maintainerRe: [PATCH 4/4] numa: introduce numa cling feature
Quoted 23 lines · 王贇
Quoted 8 lines · Peter Zijlstra
Quoted 8 lines
Thus we introduce the numa cling, which try to prevent tasks leaving the preferred node on wakeup fast path.
kernel/sched/fair.c +0
-0@@ -6195,6 +6447,13 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)if ((unsigned )i < nr_cpumask_bits)return i;Quoted 12 lines
kernel/sched/fair.c static int select_idle_sibling(struct task_struct *p, int prev, int target) · line 6458
Quoted 9 lines
kernel/sched/fair.c +7
-0+/* +* Failed to find an idle cpu, wake affine may want to pull but +* try stay on prev-cpu when the task cling to it. +*/ +if (task_numa_cling (p,cpu_to_node (prev),cpu_to_node (target))) +return prev; +return target; }Select idle sibling should never cross node boundaries and is thus the entirely wrong place to fix anything.
Quoted 3 lines
Hmm.. in our early testing the printk show both
select_task_rq_fair()andtask_numa_find_cpu()will callselect_idle_siblingwith prev and target on different node, thus we pick this point to save few lines.
But it will never return @prev if it is not in the same cache domain as @target. See how everything is gated by:
&& cpus_share_cache(x, target)
Quoted 3 lines
But if the semantics of
select_idle_sibling()is to return cpu on the same node of target, what about move the logical afterselect_idle_sibling()for the two callers?
No, that's insane. You don't do select_idle_sibling() to then ignore the result. You have to change @target before calling select_idle_sibling().
Re: [PATCH 4/4] numa: introduce numa cling feature
On 2019/7/12 下午3:53, Peter Zijlstra wrote:
[snip]
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
return target;
}Select idle sibling should never cross node boundaries and is thus the entirely wrong place to fix anything.
Quoted 8 lines
Quoted 3 lines
Hmm.. in our early testing the printk show both
select_task_rq_fair()andtask_numa_find_cpu()will callselect_idle_siblingwith prev and target on different node, thus we pick this point to save few lines.But it will never return @prev if it is not in the same cache domain as @target. See how everything is gated by:
&&
cpus_share_cache(x, target)
Yeah, that's right.
Quoted 8 lines
Quoted 4 lines
But if the semantics of
select_idle_sibling()is to return cpu on the same node of target, what about move the logical afterselect_idle_sibling()for the two callers?No, that's insane. You don't do
select_idle_sibling()to then ignore the result. You have to change @target before callingselect_idle_sibling().
I see, we should not override the decision of select_idle_sibling().
Actually the original design we try to achieve is:
let wake affine select the target
try find idle sibling of target
if got one
pick it
else if task cling to prev
pick prev
That is to consider wake affine superior to numa cling.
But after rethinking maybe this is not necessary, since numa cling is also some kind of strong wake affine hint, actually maybe even a better one to filter out the bad cases.
I'll try change @target instead and give a retest then.
Regards,
Michael Wang
Re: [PATCH 4/4] numa: introduce numa cling feature
On 2019/7/12 下午4:58, 王贇 wrote:
[snip]
Quoted 12 lines
I see, we should not override the decision of
select_idle_sibling().Actually the original design we try to achieve is:
let wake affine select the target
try find idle sibling of target
if got one
pick it
else if task cling to prev
pick prev
Quoted 7 lines
That is to consider wake affine superior to numa cling.
But after rethinking maybe this is not necessary, since numa cling is also some kind of strong wake affine hint, actually maybe even a better one to filter out the bad cases.
I'll try change @target instead and give a retest then.
We now leave select_idle_sibling() untouched, instead prevent numa swap with task cling to dst, and stop wake affine when curr & prev cpu are on different node and wakee cling to prev.
Retesting show a even better results, benchmark like dbench also show 1%~5% improvement, not stable but always improved now :-)
Regards,
Michael Wang
Quoted 4 lines
Regards,
Michael Wang
Re: [PATCH 0/4] per cgroup numa suite
In reply to 王贇Hi folks,
How do you think about these patches?
During most of our tests the results show stable improvements, thus we consider this as a generic problem and proposed this solution, hope to help address the issue.
Comments are sincerely welcome :-)
Regards,
Michael Wang
Quoted 53 lines · 王贇
During our torturing on numa stuff, we found problems like:
* missing per-cgroup information about the per-node execution status
* missing per-cgroup information about the numa localityThat is when we have a cpu cgroup running with bunch of tasks, no good way to tell how it's tasks are dealing with numa.
The first two patches are trying to complete the missing pieces, but more problems appeared after monitoring these status:
* tasks not always running on the preferred numa node
* tasks from same cgroup running on different nodesThe task numa group handler will always check if tasks are sharing pages and try to pack them into a single numa group, so they will have chance to settle down on the same node, but this failed in some cases:
* workloads share page caches rather than share mappings
* workloads got too many wakeup across nodesSince page caches are not traced by numa balancing, there are no way to realize such kind of relationship, and when there are too many wakeup, task will be drag from the preferred node and then migrate back by numa balancing, repeatedly.
Here the third patch try to address the first issue, we could now give hint to kernel about the relationship of tasks, and pack them into single numa group.
And the forth patch introduced numa cling, which try to address the wakup issue, now we try to make task stay on the preferred node on wakeup in fast path, in order to address the unbalancing risk, we monitoring the numa migration failure ratio, and pause numa cling when it reach the specified degree.
Michael Wang (4):
numa: introduce per-cgroup numa balancing locality statistic
numa: append per-node execution info in memory.numa_stat
numa: introduce numa group per task group
numa: introduce numa cling featureinclude/linux/memcontrol.h | 37 ++++ include/linux/sched.h | 8 +- include/linux/sched/sysctl.h | 3 + kernel/sched/core.c | 37 ++++ kernel/sched/debug.c | 7 + kernel/sched/fair.c | 455 ++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 14 ++ kernel/sysctl.c | 9 + mm/memcontrol.c | 66 +++++++ 9 files changed, 628 insertions(+), 8 deletions(-)
v2 cover per-cgroup numa suite
In reply to 王贇During our torturing on numa stuff, we found problems like:
* missing per-cgroup information about the per-node execution status
* missing per-cgroup information about the numa locality
That is when we have a cpu cgroup running with bunch of tasks, no good way to tell how it's tasks are dealing with numa.
The first two patches are trying to complete the missing pieces, but more problems appeared after monitoring these status:
* tasks not always running on the preferred numa node
* tasks from same cgroup running on different nodes
The task numa group handler will always check if tasks are sharing pages and try to pack them into a single numa group, so they will have chance to settle down on the same node, but this failed in some cases:
* workloads share page caches rather than share mappings
* workloads got too many wakeup across nodes
Since page caches are not traced by numa balancing, there are no way to realize such kind of relationship, and when there are too many wakeup, task will be drag from the preferred node and then migrate back by numa balancing, repeatedly.
Here the third patch try to address the first issue, we could now give hint to kernel about the relationship of tasks, and pack them into single numa group.
And the forth patch introduced numa cling, which try to address the wakup issue, now we try to make task stay on the preferred node on wakeup in fast path, in order to address the unbalancing risk, we monitoring the numa migration failure ratio, and pause numa cling when it reach the specified degree.
Since v1:
* move statistics from memory cgroup into cpu group
* statistics now accounting in hierarchical way
* locality now accounted into 8 regions equally
* numa cling no longer override select_idle_sibling, instead we prevent numa swap migration with tasks cling to dst-node, also prevent wake affine to drag tasks away which already cling to prev-cpu
* other refine on comments and names
Michael Wang (4):
v2 numa: introduce per-cgroup numa balancing locality statistic v2 numa: append per-node execution time in cpu.numa_stat v2 numa: introduce numa group per task group
v4 numa: introduce numa cling feature
include/linux/sched.h | 8 +- include/linux/sched/sysctl.h | 3 + kernel/sched/core.c | 85 ++++++++ kernel/sched/debug.c | 7 + kernel/sched/fair.c | 510 ++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 41 ++++ kernel/sysctl.c | 9 + 7 files changed, 651 insertions(+), 12 deletions(-)
Signature · 1 line
2.14.4.44.g2045bb6
PATCH v2 1/4 numa: introduce per-cgroup numa balancing locality statistic
This patch introduced numa locality statistic, which try to imply the numa balancing efficiency per memory cgroup.
On numa balancing, we trace the local page accessing ratio of tasks, which we call the locality.
By doing 'cat /sys/fs/cgroup/cpu/CGROUP_PATH/cpu.numa_stat', we see output line heading with 'locality', like:
locality 15393 21259 13023 44461 21247 17012 28496 145402
locality divided into 8 regions, each number standing for the micro seconds we hit a task running with the locality within that region, for example here we have tasks with locality around 0~12% running for 15393 ms, and tasks with locality around 88~100% running for 145402 ms.
By monitoring the increment, we can check if the workloads of a particular cgroup is doing well with numa, when most of the tasks are running in low locality region, then something is wrong with your numa policy.
---
Since v1:
* move implementation from memory cgroup into cpu group
* introduce new entry 'numa_stat' to present locality
* locality now accounting in hierarchical way
* locality now accounted into 8 regions equally
include/linux/sched.h | 8 +++++++- kernel/sched/core.c | 40 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/debug.c | 7 +++++++ kernel/sched/fair.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 29 +++++++++++++++++++++++++++++ 5 files changed, 132 insertions(+), 1 deletion(-)
include/linux/sched.h +7 -1
@@ -1117,8 +1117,14 @@ struct task_struct {* scan window were remote/local or failed to migrate. The task scan * period is adapted based on the locality of the faults with different * weights depending on whether they were shared or private faults +* +* 0 -- remote faults +* 1 -- local faults +* 2 -- page migration failure +* 3 -- remote page accessing +* 4 -- local page accessing */ -unsigned long numa_faults_locality[3 ]; +unsigned long numa_faults_locality[5 ];unsigned long numa_pages_migrated;#endif /* CONFIG_NUMA_BALANCING */
kernel/sched/core.c +40 -0
@@ -6367,6 +6367,10 @@ static struct kmem_cache *task_group_cache __read_mostly;DECLARE_PER_CPU (cpumask_var_t, load_balance_mask);DECLARE_PER_CPU (cpumask_var_t, select_idle_mask); +#ifdef CONFIG_NUMA_BALANCING +DECLARE_PER_CPU(struct numa_stat, root_numa_stat); +#endif +void __init sched_init (void ) {unsigned long alloc_size =0 , ptr;
@@ -6416,6 +6420,10 @@ void __init sched_init(void)init_defrootdomain ();#endif +#ifdef CONFIG_NUMA_BALANCING + root_task_group.numa_stat = &root_numa_stat; +#endif +#ifdef CONFIG_RT_GROUP_SCHED init_rt_bandwidth (&root_task_group.rt_bandwidth,global_rt_period (),global_rt_runtime ());
@@ -6727,6 +6735,7 @@ static DEFINE_SPINLOCK(task_group_lock);static void sched_free_group (struct task_group *tg) { +free_tg_numa_stat (tg);free_fair_sched_group (tg);free_rt_sched_group (tg);autogroup_free (tg);
@@ -6742,6 +6751,9 @@ struct task_group *sched_create_group(struct task_group *parent)if (!tg)return ERR_PTR (-ENOMEM ); +if (!alloc_tg_numa_stat (tg)) + goto err; +if (!alloc_fair_sched_group (tg, parent)) goto err;
@@ -7277,6 +7289,28 @@ static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, }#endif /* CONFIG_RT_GROUP_SCHED */ +#ifdef CONFIG_NUMA_BALANCING +static int cpu_numa_stat_show (struct seq_file *sf,void *v) +{ +int nr; +struct task_group *tg =css_tg (seq_css (sf)); + +seq_puts (sf,"locality" ); +for (nr =0 ; nr <NR_NL_INTERVAL ; nr++) { +int cpu; +u64 sum =0 ; + +for_each_possible_cpu (cpu) + sum +=per_cpu (tg->numa_stat->locality[nr], cpu); + +seq_printf (sf," %u" ,jiffies_to_msecs (sum)); + } +seq_putc (sf,'\n' ); + +return 0 ; +} +#endif + static struct cftype cpu_legacy_files[] = {#ifdef CONFIG_FAIR_GROUP_SCHED {
@@ -7312,6 +7346,12 @@ static struct cftype cpu_legacy_files[] = { .read_u64 = cpu_rt_period_read_uint, .write_u64 = cpu_rt_period_write_uint, }, +#endif +#ifdef CONFIG_NUMA_BALANCING + { + .name = "numa_stat", + .seq_show = cpu_numa_stat_show, + }, #endif { }/* Terminate */ };
kernel/sched/debug.c +7 -0
@@ -848,6 +848,13 @@ static void sched_show_numa(struct task_struct *p, struct seq_file *m)P (total_numa_faults);SEQ_printf (m,"current_node=%d, numa_group_id=%d\n" ,task_node (p),task_numa_group_id (p)); +SEQ_printf (m,"faults_locality local=%lu remote=%lu failed=%lu " , + p->numa_faults_locality[1 ], + p->numa_faults_locality[0 ], + p->numa_faults_locality[2 ]); +SEQ_printf (m,"lhit=%lu rhit=%lu\n" , + p->numa_faults_locality[4 ], + p->numa_faults_locality[3 ]);show_numa_stats (p, m);mpol_put (pol);#endif
kernel/sched/fair.c +49 -0
67 lines not on this page · raw message
kernel/sched/sched.h +29 -0
@@ -353,6 +353,17 @@ struct cfs_bandwidth {#endif }; +#ifdef CONFIG_NUMA_BALANCING + +/* NUMA Locality Interval, 8 bucket for cache align */ +#define NR_NL_INTERVAL 8 + +struct numa_stat { +u64 locality[NR_NL_INTERVAL ]; +}; + +#endif +/* Task group related information */ struct task_group {struct cgroup_subsys_state css;
@@ -393,8 +404,26 @@ struct task_group {#endif struct cfs_bandwidth cfs_bandwidth; + +#ifdef CONFIG_NUMA_BALANCING +struct numa_stat __percpu *numa_stat; +#endif }; +#ifdef CONFIG_NUMA_BALANCING +int alloc_tg_numa_stat (struct task_group *tg); +void free_tg_numa_stat (struct task_group *tg); +#else +static int alloc_tg_numa_stat (struct task_group *tg) +{ +return 1 ; +} + +static void free_tg_numa_stat (struct task_group *tg) +{ +} +#endif +#ifdef CONFIG_FAIR_GROUP_SCHED #define ROOT_TASK_GROUP_LOAD NICE_0_LOAD
Signature · 1 line
2.14.4.44.g2045bb6
PATCH v2 2/4 numa: append per-node execution time in cpu.numa_stat
In reply to 王贇This patch introduced numa execution time information, to imply the numa efficiency.
By doing 'cat /sys/fs/cgroup/cpu/CGROUP_PATH/cpu.numa_stat', we see new output line heading with 'exectime', like:
exectime 311900 407166
which means the tasks of this cgroup executed 311900 micro seconds on node 0, and 407166 ms on node 1.
Combined with the memory node info from memory cgroup, we can estimate the numa efficiency, for example if the memory.numa_stat show:
total=206892 N0=21933 N1=185171
By monitoring the increments, if the topology keep in this way and locality is not nice, then it imply numa balancing can't help migrate the memory from node 1 to 0 which is accessing by tasks on node 0, or tasks can't migrate to node 1 for some reason, then you may consider to bind the workloads on the cpus of node 1.
---
Since v1:
* move implementation from memory cgroup into cpu group
* exectime now accounting in hierarchical way
* change member name into jiffies
kernel/sched/core.c | 12 ++++++++++++ kernel/sched/fair.c | 2 ++ kernel/sched/sched.h | 1 + 3 files changed, 15 insertions(+)
kernel/sched/core.c +12 -0
@@ -7307,6 +7307,18 @@ static int cpu_numa_stat_show(struct seq_file *sf, void *v) }seq_putc (sf,'\n' ); +seq_puts (sf,"exectime" ); +for_each_online_node (nr) { +int cpu; +u64 sum =0 ; + +for_each_cpu (cpu,cpumask_of_node (nr)) + sum +=per_cpu (tg->numa_stat->jiffies, cpu); + +seq_printf (sf," %u" ,jiffies_to_msecs (sum)); + } +seq_putc (sf,'\n' ); +return 0 ; }#endif
kernel/sched/fair.c +2 -0
@@ -2652,6 +2652,8 @@ static void update_tg_numa_stat(struct task_struct *p)if (idx !=-1 )this_cpu_inc (tg->numa_stat->locality[idx]); +this_cpu_inc (tg->numa_stat->jiffies); + tg = tg->parent; }
kernel/sched/sched.h +1 -0
@@ -360,6 +360,7 @@ struct cfs_bandwidth {struct numa_stat {u64 locality[NR_NL_INTERVAL ]; +u64 jiffies; };#endif
Signature · 1 line
2.14.4.44.g2045bb6
Michal Koutný
maintainerRe: [PATCH v2 2/4] numa: append per-node execution time in cpu.numa_stat
Quoted 4 lines · 王贇
By doing 'cat
/sys/fs/cgroup/cpu/CGROUP_PATH/cpu.numa_stat', we see new output line heading with 'exectime', like:exectime 311900 407166
What you present are times aggregated over CPUs in the NUMA nodes, this seems a bit lossy interface.
Despite you the aggregated information is sufficient for your monitoring, I think it's worth providing the information with the original granularity.
Note that cpuacct v1 controller used to report such percpu runtime stats. The v2 implementation would rather build upon the rstat API.
Michal
Re: [PATCH v2 2/4] numa: append per-node execution time in cpu.numa_stat
Quoted 11 lines · Michal Koutný
Quoted 4 lines · 王贇
By doing 'cat
/sys/fs/cgroup/cpu/CGROUP_PATH/cpu.numa_stat', we see new output line heading with 'exectime', like:exectime 311900 407166
What you present are times aggregated over CPUs in the NUMA nodes, this seems a bit lossy interface.
Despite you the aggregated information is sufficient for your monitoring, I think it's worth providing the information with the original granularity.
As Peter suggested previously, kernel do not report jiffies to user anymore and 'ms' could be better, I guess usually we care about how much the percentage is on a particular node?
Quoted 3 lines
Note that cpuacct v1 controller used to report such percpu runtime stats. The v2 implementation would rather build upon the rstat API.
Support cgroup v2 is on the plan :-) let's mark this as todo currently, i suppose they may not share the same piece of code.
Regards,
Michael Wang
Quoted 3 lines
Michal
PATCH v2 3/4 numa: introduce numa group per task group
In reply to 王贇By tracing numa page faults, we recognize tasks sharing the same page, and try pack them together into a single numa group.
However when two task share lot's of cache pages while not much anonymous pages, since numa balancing do not tracing cache page, they have no chance to join into the same group.
While tracing cache page cost too much, we could use some hints from userland and cpu cgroup could be a good one.
This patch introduced new entry 'numa_group' for cpu cgroup, by echo non-zero into the entry, we can now force all the tasks of this cgroup to join the same numa group serving for task group.
In this way tasks are more likely to settle down on the same node, to share closer cpu cache and gain benefit from NUMA on both file/anonymous pages.
Besides, when multiple cgroup enabled numa group, they will be able to exchange task location by utilizing numa migration, in this way they could achieve single node settle down without breaking load balance.
---
Since v1:
* just rebase, no logical changes
kernel/sched/core.c | 33 ++++++++++ kernel/sched/fair.c | 175 ++++++++++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 11 ++++ 3 files changed, 218 insertions(+), 1 deletion(-)
kernel/sched/core.c +33 -0
@@ -6802,6 +6802,8 @@ void sched_offline_group(struct task_group *tg) {unsigned long flags; +update_tg_numa_group (tg, false); +/* End participation in shares distribution: */ unregister_fair_sched_group (tg);
@@ -7321,6 +7323,32 @@ static int cpu_numa_stat_show(struct seq_file *sf, void *v)return 0 ; } + +static DEFINE_MUTEX (numa_mutex); + +static int cpu_numa_group_show (struct seq_file *sf,void *v) +{ +struct task_group *tg =css_tg (seq_css (sf)); + +mutex_lock (&numa_mutex); +show_tg_numa_group (tg, sf); +mutex_unlock (&numa_mutex); + +return 0 ; +} + +static int cpu_numa_group_write_s64 (struct cgroup_subsys_state *css, +struct cftype *cft,s64 numa_group) +{ +int ret; +struct task_group *tg =css_tg (css); + +mutex_lock (&numa_mutex); + ret =update_tg_numa_group (tg, numa_group); +mutex_unlock (&numa_mutex); + +return ret; +}#endif static struct cftype cpu_legacy_files[] = {
@@ -7364,6 +7392,11 @@ static struct cftype cpu_legacy_files[] = { .name = "numa_stat", .seq_show = cpu_numa_stat_show, }, + { + .name = "numa_group", + .write_s64 = cpu_numa_group_write_s64, + .seq_show = cpu_numa_group_show, + },#endif { }/* Terminate */ };
kernel/sched/fair.c +174 -1
193 lines not on this page · raw message
kernel/sched/sched.h +11 -0
@@ -408,6 +408,7 @@ struct task_group {#ifdef CONFIG_NUMA_BALANCING struct numa_stat __percpu *numa_stat; +void *numa_group;#endif };
@@ -1316,11 +1317,21 @@ extern int migrate_task_to(struct task_struct *p, int cpu);extern int migrate_swap (struct task_struct *p,struct task_struct *t,int cpu,int scpu);extern void init_numa_balancing (unsigned long clone_flags,struct task_struct *p); +extern void show_tg_numa_group (struct task_group *tg,struct seq_file *sf); +extern int update_tg_numa_group (struct task_group *tg,bool numa_group);#else static inline void init_numa_balancing (unsigned long clone_flags,struct task_struct *p) { } +static inline void +show_tg_numa_group (struct task_group *tg,struct seq_file *sf) +{ +} +update_tg_numa_group(struct task_group *tg, bool numa_group) +{ +return 0 ; +}#endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_SMP
Signature · 1 line
2.14.4.44.g2045bb6
PATCH v4 4/4 numa: introduce numa cling feature
In reply to 王贇Although we paid so many effort to settle down task on a particular node, there are still chances for a task to leave it's preferred node, that is by wakeup, numa swap migrations or load balance.
When we are using cpu cgroup in share way, since all the workloads see all the cpus, it could be really bad especially when there are too many fast wakeup, although now we can numa group the tasks, they won't really stay on the same node, for example we have numa group ng_A, ng_B, ng_C, ng_D, it's very likely result as:
CPU Usage: Node 0 Node 1 ng_A(600%) ng_A(400%) ng_B(400%) ng_B(600%) ng_C(400%) ng_C(600%) ng_D(600%) ng_D(400%) Memory Ratio: Node 0 Node 1 ng_A(60%) ng_A(40%) ng_B(40%) ng_B(60%) ng_C(40%) ng_C(60%) ng_D(60%) ng_D(40%)
Locality won't be too bad but far from the best situation, we want a numa group to settle down thoroughly on a particular node, with every thing balanced.
Thus we introduce the numa cling, which try to prevent tasks leaving the preferred node on wakeup fast path.
This help thoroughly settle down the workloads on single node, but when multiple numa group try to settle down on the same node, unbalancing could happen.
For example we have numa group ng_A, ng_B, ng_C, ng_D, it may result in situation like:
CPU Usage:
Node 0 Node 1 ng_A(1000%) ng_B(1000%) ng_C(400%) ng_C(600%) ng_D(400%) ng_D(600%)
Memory Ratio:
Node 0 Node 1 ng_A(100%) ng_B(100%) ng_C(10%) ng_C(90%) ng_D(10%) ng_D(90%)
This is because when ng_C, ng_D start to have most of the memory on node 1 at some point, task_x of ng_C stay on node 0 will try to do numa swap migration with the task_y of ng_D stay on node 1 as long as load balanced, the result is task_x stay on node 1 and task_y stay on node 0, while both of them prefer node 1.
Now when other tasks of ng_D stay on node 1 wakeup task_y, task_y will very likely go back to node 1, and since numa cling enabled, it will keep stay on node 1 although load unbalanced, this could be frequently and more and more tasks will prefer the node 1 and make it busy.
So the key point here is to stop doing numa cling when load starting to become unbalancing.
We achieved this by monitoring the migration failure ratio, in scenery above, too much tasks prefer node 1 and will keep migrating to it, load unbalancing could lead into the migration failure in this case, and when the failure ratio above the specified degree, we pause the cling and try to resettle the workloads on a better node by stop tasks prefer the busy node, this will finally give us the result like:
CPU Usage:
Node 0 Node 1 ng_A(1000%) ng_B(1000%) ng_C(1000%) ng_D(1000%)
Memory Ratio:
Node 0 Node 1 ng_A(100%) ng_B(100%) ng_C(100%) ng_D(100%)
Now we achieved the best locality and maximum hot cache benefit.
Tested on a 2 node box with 96 cpus, do sysbench-mysql-oltp_read_write testing, X mysqld instances created and attached to X cgroups, X sysbench instances then created and attached to corresponding cgroup to test the mysql with oltp_read_write script for 20 minutes, average eps show:
origin ng + cling
4 instances each 24 threads 7641.27 8010.18 +4.83%
4 instances each 48 threads 9423.39 10021.03 +6.34%
4 instances each 72 threads 9691.47 10192.73 +5.17%
8 instances each 24 threads 4485.44 4577.95 +2.06%
8 instances each 48 threads 5565.06 5737.50 +3.10%
8 instances each 72 threads 5605.20 5752.33 +2.63%
Also tested with perf-bench-numa, dbench, sysbench-memory, pgbench, tiny improvement observed.
---
Since v3:
* numa cling no longer override select_idle_sibling, instead we prevent numa swap migration with tasks cling to dst-node, also prevent wake affine to drag tasks away which already cling to prev-cpu
* refine comments
include/linux/sched/sysctl.h | 3 + kernel/sched/fair.c | 296 ++++++++++++++++++++++++++++++++++++++++--- kernel/sysctl.c | 9 ++ 3 files changed, 292 insertions(+), 16 deletions(-)
include/linux/sched/sysctl.h +3 -0
@@ -38,6 +38,9 @@ extern unsigned int sysctl_numa_balancing_scan_period_min;extern unsigned int sysctl_numa_balancing_scan_period_max;extern unsigned int sysctl_numa_balancing_scan_size; +extern unsigned int sysctl_numa_balancing_cling_degree; +extern unsigned int max_numa_balancing_cling_degree; +#ifdef CONFIG_SCHED_DEBUG extern __read_mostly unsigned int sysctl_sched_migration_cost;extern __read_mostly unsigned int sysctl_sched_nr_migrate;
kernel/sched/fair.c +280 -16
437 lines not on this page · raw message
kernel/sysctl.c +9 -0
@@ -417,6 +417,15 @@ static struct ctl_table kern_table[] = { .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ONE, }, + { + .procname = "numa_balancing_cling_degree", + .data = &sysctl_numa_balancing_cling_degree, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = &max_numa_balancing_cling_degree, + }, { .procname = "numa_balancing", .data = NULL, /* filled in by handler */
Signature · 1 line
2.14.4.44.g2045bb6
PATCH v5 4/4 numa: introduce numa cling feature
Although we paid so many effort to settle down task on a particular node, there are still chances for a task to leave it's preferred node, that is by wakeup, numa swap migrations or load balance.
When we are using cpu cgroup in share way, since all the workloads see all the cpus, it could be really bad especially when there are too many fast wakeup, although now we can numa group the tasks, they won't really stay on the same node, for example we have numa group ng_A, ng_B, ng_C, ng_D, it's very likely result as:
CPU Usage: Node 0 Node 1 ng_A(600%) ng_A(400%) ng_B(400%) ng_B(600%) ng_C(400%) ng_C(600%) ng_D(600%) ng_D(400%) Memory Ratio: Node 0 Node 1 ng_A(60%) ng_A(40%) ng_B(40%) ng_B(60%) ng_C(40%) ng_C(60%) ng_D(60%) ng_D(40%)
Locality won't be too bad but far from the best situation, we want a numa group to settle down thoroughly on a particular node, with every thing balanced.
Thus we introduce the numa cling, which try to prevent tasks leaving the preferred node on wakeup fast path.
This help thoroughly settle down the workloads on single node, but when multiple numa group try to settle down on the same node, unbalancing could happen.
For example we have numa group ng_A, ng_B, ng_C, ng_D, it may result in situation like:
CPU Usage:
Node 0 Node 1 ng_A(1000%) ng_B(1000%) ng_C(400%) ng_C(600%) ng_D(400%) ng_D(600%)
Memory Ratio:
Node 0 Node 1 ng_A(100%) ng_B(100%) ng_C(10%) ng_C(90%) ng_D(10%) ng_D(90%)
This is because when ng_C, ng_D start to have most of the memory on node 1 at some point, task_x of ng_C stay on node 0 will try to do numa swap migration with the task_y of ng_D stay on node 1 as long as load balanced, the result is task_x stay on node 1 and task_y stay on node 0, while both of them prefer node 1.
Now when other tasks of ng_D stay on node 1 wakeup task_y, task_y will very likely go back to node 1, and since numa cling enabled, it will keep stay on node 1 although load unbalanced, this could be frequently and more and more tasks will prefer the node 1 and make it busy.
So the key point here is to stop doing numa cling when load starting to become unbalancing.
We achieved this by monitoring the migration failure ratio, in scenery above, too much tasks prefer node 1 and will keep migrating to it, load unbalancing could lead into the migration failure in this case, and when the failure ratio above the specified degree, we pause the cling and try to resettle the workloads on a better node by stop tasks prefer the busy node, this will finally give us the result like:
CPU Usage:
Node 0 Node 1 ng_A(1000%) ng_B(1000%) ng_C(1000%) ng_D(1000%)
Memory Ratio:
Node 0 Node 1 ng_A(100%) ng_B(100%) ng_C(100%) ng_D(100%)
Now we achieved the best locality and maximum hot cache benefit.
Tested on a 2 node box with 96 cpus, do sysbench-mysql-oltp_read_write testing, X mysqld instances created and attached to X cgroups, X sysbench instances then created and attached to corresponding cgroup to test the mysql with oltp_read_write script for 20 minutes, average eps show:
origin ng + cling
4 instances each 24 threads 7641.27 8010.18 +4.83%
4 instances each 48 threads 9423.39 10021.03 +6.34%
4 instances each 72 threads 9691.47 10192.73 +5.17%
8 instances each 24 threads 4485.44 4577.95 +2.06%
8 instances each 48 threads 5565.06 5737.50 +3.10%
8 instances each 72 threads 5605.20 5752.33 +2.63%
Also tested with perf-bench-numa, dbench, sysbench-memory, pgbench, tiny improvement observed.
---
Since v4:
* Trivial cleanup
include/linux/sched/sysctl.h | 3 + kernel/sched/fair.c | 294 ++++++++++++++++++++++++++++++++++++++++--- kernel/sysctl.c | 9 ++ 3 files changed, 291 insertions(+), 15 deletions(-)
include/linux/sched/sysctl.h +3 -0
@@ -38,6 +38,9 @@ extern unsigned int sysctl_numa_balancing_scan_period_min;extern unsigned int sysctl_numa_balancing_scan_period_max;extern unsigned int sysctl_numa_balancing_scan_size; +extern unsigned int sysctl_numa_balancing_cling_degree; +extern unsigned int max_numa_balancing_cling_degree; +#ifdef CONFIG_SCHED_DEBUG extern __read_mostly unsigned int sysctl_sched_migration_cost;extern __read_mostly unsigned int sysctl_sched_nr_migrate;
kernel/sched/fair.c +279 -15
424 lines not on this page · raw message
kernel/sysctl.c +9 -0
@@ -417,6 +417,15 @@ static struct ctl_table kern_table[] = { .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ONE, }, + { + .procname = "numa_balancing_cling_degree", + .data = &sysctl_numa_balancing_cling_degree, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = &max_numa_balancing_cling_degree, + }, { .procname = "numa_balancing", .data = NULL, /* filled in by handler */
Signature · 1 line
2.14.4.44.g2045bb6
Re: [PATCH v2 0/4] per-cgroup numa suite
In reply to 王贇Hi, Peter
Now we have all these stuff in cpu cgroup, with the new statistic folks should be able to estimate their per-cgroup workloads on numa platform, and numa group + cling would help to address the issue when their workloads can't be settled on one node.
How do you think about this version :-)
Regards,
Michael Wang
Quoted 61 lines · 王贇
During our torturing on numa stuff, we found problems like:
* missing per-cgroup information about the per-node execution status
* missing per-cgroup information about the numa localityThat is when we have a cpu cgroup running with bunch of tasks, no good way to tell how it's tasks are dealing with numa.
The first two patches are trying to complete the missing pieces, but more problems appeared after monitoring these status:
* tasks not always running on the preferred numa node
* tasks from same cgroup running on different nodesThe task numa group handler will always check if tasks are sharing pages and try to pack them into a single numa group, so they will have chance to settle down on the same node, but this failed in some cases:
* workloads share page caches rather than share mappings
* workloads got too many wakeup across nodesSince page caches are not traced by numa balancing, there are no way to realize such kind of relationship, and when there are too many wakeup, task will be drag from the preferred node and then migrate back by numa balancing, repeatedly.
Here the third patch try to address the first issue, we could now give hint to kernel about the relationship of tasks, and pack them into single numa group.
And the forth patch introduced numa cling, which try to address the wakup issue, now we try to make task stay on the preferred node on wakeup in fast path, in order to address the unbalancing risk, we monitoring the numa migration failure ratio, and pause numa cling when it reach the specified degree.
Since v1:
* move statistics from memory cgroup into cpu group
* statistics now accounting in hierarchical way
* locality now accounted into 8 regions equally
* numa cling no longer overrideselect_idle_sibling, instead we prevent numa swap migration with tasks cling to dst-node, also prevent wake affine to drag tasks away which already cling to prev-cpu
* other refine on comments and namesMichael Wang (4):
v2 numa: introduce per-cgroup numa balancing locality statistic v2 numa: append per-node execution time in cpu.numa_stat v2 numa: introduce numa group per task group
v4 numa: introduce numa cling featureinclude/linux/sched.h | 8 +- include/linux/sched/sysctl.h | 3 + kernel/sched/core.c | 85 ++++++++ kernel/sched/debug.c | 7 + kernel/sched/fair.c | 510 ++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 41 ++++ kernel/sysctl.c | 9 + 7 files changed, 651 insertions(+), 12 deletions(-)
Re: [PATCH v2 0/4] per-cgroup numa suite
In reply to 王贇Hi, Folks
Please feel free to comment if you got any concerns :-)
Hi, Peter
How do you think about this version?
Please let us know if it's still not good enough to be accepted :-)
Regards,
Michael Wang
Quoted 61 lines · 王贇
During our torturing on numa stuff, we found problems like:
* missing per-cgroup information about the per-node execution status
* missing per-cgroup information about the numa localityThat is when we have a cpu cgroup running with bunch of tasks, no good way to tell how it's tasks are dealing with numa.
The first two patches are trying to complete the missing pieces, but more problems appeared after monitoring these status:
* tasks not always running on the preferred numa node
* tasks from same cgroup running on different nodesThe task numa group handler will always check if tasks are sharing pages and try to pack them into a single numa group, so they will have chance to settle down on the same node, but this failed in some cases:
* workloads share page caches rather than share mappings
* workloads got too many wakeup across nodesSince page caches are not traced by numa balancing, there are no way to realize such kind of relationship, and when there are too many wakeup, task will be drag from the preferred node and then migrate back by numa balancing, repeatedly.
Here the third patch try to address the first issue, we could now give hint to kernel about the relationship of tasks, and pack them into single numa group.
And the forth patch introduced numa cling, which try to address the wakup issue, now we try to make task stay on the preferred node on wakeup in fast path, in order to address the unbalancing risk, we monitoring the numa migration failure ratio, and pause numa cling when it reach the specified degree.
Since v1:
* move statistics from memory cgroup into cpu group
* statistics now accounting in hierarchical way
* locality now accounted into 8 regions equally
* numa cling no longer overrideselect_idle_sibling, instead we prevent numa swap migration with tasks cling to dst-node, also prevent wake affine to drag tasks away which already cling to prev-cpu
* other refine on comments and namesMichael Wang (4):
v2 numa: introduce per-cgroup numa balancing locality statistic v2 numa: append per-node execution time in cpu.numa_stat v2 numa: introduce numa group per task group
v4 numa: introduce numa cling featureinclude/linux/sched.h | 8 +- include/linux/sched/sysctl.h | 3 + kernel/sched/core.c | 85 ++++++++ kernel/sched/debug.c | 7 + kernel/sched/fair.c | 510 ++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 41 ++++ kernel/sysctl.c | 9 + 7 files changed, 651 insertions(+), 12 deletions(-)