fix unsigned pcp adjustments
Subsystems (6)
v2 cover fix unsigned pcp adjustments
As of v3.11-9444-g3ea67d0 "memcg: add per cgroup writeback pages accounting" memcg use of __this_cpu_add(counter, -nr_pages) leads to incorrect statistic values because the negated nr_pages is not sign extended (counter is long, nr_pages is unsigned int). The memcg fix is __this_cpu_sub(counter, nr_pages). But that doesn't simply work because __this_cpu_sub(counter, nr_pages) was implemented as __this_cpu_add(counter, -nr_pages) which suffers the same problem. Example:
unsigned int delta = 1
preempt_disable()
this_cpu_write(long_counter, 0)
this_cpu_sub(long_counter, delta)
preempt_enable()
Before this change long_counter on a 64 bit machine ends with value 0xffffffff, rather than 0xffffffffffffffff. This is because this_cpu_sub(pcp, delta) boils down to:
long_counter = 0 + 0xffffffff
v3.12-rc6 shows that only new memcg code is affected by this problem - the new mem_cgroup_move_account_page_stat() is the only place where an unsigned adjustment is used. All other callers (e.g. shrink_dcache_sb) already use a signed adjustment, so no problems before v3.12. Though I did not audit the stable kernel trees, so there could be something hiding in there.
Patch 1 creates a test module for percpu operations which demonstrates the __this_cpu_sub() problems. This patch is independent can be discarded if there is no interest.
Patch 2 fixes __this_cpu_sub() to work with unsigned adjustments.
Patch 3 uses __this_cpu_sub() in memcg.
An alternative smaller solution is for memcg to use:
__this_cpu_add(counter, -(int)nr_pages)
admitting that __this_cpu_add/sub() doesn't work with unsigned adjustments. But I felt like fixing the core services to prevent this in the future.
Changes from V1:
- more accurate patch titles, patch logs, and test module description now referring to per cpu operations rather than per cpu counters.
- move small test code update from patch 2 to patch 1 (where the test is introduced).
Greg Thelen (3):
percpu: add test module for various percpu operations
percpu: fix this_cpu_sub() subtrahend casting for unsigneds
memcg: use __this_cpu_sub() to dec stats to avoid incorrect subtrahend
casting
arch/x86/include/asm/percpu.h | 3 +- include/linux/percpu.h | 8 +-- lib/Kconfig.debug | 9 +++ lib/Makefile | 2 + lib/percpu_test.c | 138 ++++++++++++++++++++++++++++++++++++++++++ mm/memcontrol.c | 2 +- 6 files changed, 156 insertions(+), 6 deletions(-) create mode 100644 lib/percpu_test.c
Signature · 1 line
1.8.4.1
PATCH v2 1/3 percpu: add test module for various percpu operations
Tests various percpu operations.
Enable with CONFIG_PERCPU_TEST=m.
--- lib/Kconfig.debug | 9 ++++ lib/Makefile | 2 + lib/percpu_test.c | 138 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 149 insertions(+) create mode 100644 lib/percpu_test.c
lib/Kconfig.debug +9 -0
@@ -1472,6 +1472,15 @@ config INTERVAL_TREE_TEST help A benchmark measuring the performance of the interval tree library +config PERCPU_TEST + tristate "Per cpu operations test" + depends on m && DEBUG_KERNEL + help + Enable this option to build test module which validates per-cpu + operations. + + If unsure, say N. + config ATOMIC64_SELFTEST bool "Perform an atomic64_t self-test at boot" help
lib/Makefile +2 -0
@@ -157,6 +157,8 @@ obj-$(CONFIG_INTERVAL_TREE_TEST) += interval_tree_test.o interval_tree_test-objs := interval_tree_test_main.o interval_tree.o +obj-$(CONFIG_PERCPU_TEST) += percpu_test.o + obj-$(CONFIG_ASN1) += asn1_decoder.o obj-$(CONFIG_FONT_SUPPORT) += fonts/
lib/percpu_test.c new file +138 -0
138 lines not on this page · raw message
Signature · 7 lines
1.8.4.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerRe: [PATCH v2 1/3] percpu: add test module for various percpu operations
Quoted 1 line · On Sun, 27 Oct 2013 10:30:15 -0700 Greg Thelen…
Tests various percpu operations.
Could you please take a look at the 32-bit build (this is i386):
lib/percpu_test.c: In function 'percpu_test_init':lib/percpu_test.c:61: warning: integer constant is too large for 'long' type lib/percpu_test.c:61: warning: integer constant is too large for 'long' type lib/percpu_test.c:61: warning: integer constant is too large for 'long' type lib/percpu_test.c:61: warning: integer constant is too large for 'long' type lib/percpu_test.c:61: warning: integer constant is too large for 'long' type lib/percpu_test.c:61: warning: integer constant is too large for 'long' type lib/percpu_test.c:70: warning: integer constant is too large for 'long' type lib/percpu_test.c:70: warning: integer constant is too large for 'long' type lib/percpu_test.c:70: warning: integer constant is too large for 'long' type lib/percpu_test.c:70: warning: integer constant is too large for 'long' type lib/percpu_test.c:70: warning: integer constant is too large for 'long' type lib/percpu_test.c:70: warning: integer constant is too large for 'long' type lib/percpu_test.c:89: warning: integer constant is too large for 'long' type lib/percpu_test.c:89: warning: integer constant is too large for 'long' type lib/percpu_test.c:89: warning: integer constant is too large for 'long' type lib/percpu_test.c:89: warning: integer constant is too large for 'long' type lib/percpu_test.c:89: warning: integer constant is too large for 'long' type lib/percpu_test.c:89: warning: integer constant is too large for 'long' type lib/percpu_test.c:97: warning: integer constant is too large for 'long' type lib/percpu_test.c:97: warning: integer constant is too large for 'long' type lib/percpu_test.c:97: warning: integer constant is too large for 'long' type lib/percpu_test.c:97: warning: integer constant is too large for 'long' type lib/percpu_test.c:97: warning: integer constant is too large for 'long' type lib/percpu_test.c:97: warning: integer constant is too large for 'long' type lib/percpu_test.c:112: warning: integer constant is too large for 'long' type lib/percpu_test.c:112: warning: integer constant is too large for 'long' type lib/percpu_test.c:112: warning: integer constant is too large for 'long' type lib/percpu_test.c:112: warning: integer constant is too large for 'long' type lib/percpu_test.c:112: warning: integer constant is too large for 'long' type lib/percpu_test.c:112: warning: integer constant is too large for 'long' type
Greg Thelen
Acked-by Tejun HeoRe: [PATCH v2 1/3] percpu: add test module for various percpu operations
Quoted 29 lines · Andrew Morton
Quoted 1 line · On Sun, 27 Oct 2013 10:30:15 -0700 Greg Thelen…
Tests various percpu operations.
Could you please take a look at the 32-bit build (this is i386):
lib/percpu_test.c: In function 'percpu_test_init':lib/percpu_test.c:61: warning: integer constant is too large for 'long' typelib/percpu_test.c:61: warning: integer constant is too large for 'long' typelib/percpu_test.c:61: warning: integer constant is too large for 'long' typelib/percpu_test.c:61: warning: integer constant is too large for 'long' typelib/percpu_test.c:61: warning: integer constant is too large for 'long' typelib/percpu_test.c:61: warning: integer constant is too large for 'long' typelib/percpu_test.c:70: warning: integer constant is too large for 'long' typelib/percpu_test.c:70: warning: integer constant is too large for 'long' typelib/percpu_test.c:70: warning: integer constant is too large for 'long' typelib/percpu_test.c:70: warning: integer constant is too large for 'long' typelib/percpu_test.c:70: warning: integer constant is too large for 'long' typelib/percpu_test.c:70: warning: integer constant is too large for 'long' typelib/percpu_test.c:89: warning: integer constant is too large for 'long' typelib/percpu_test.c:89: warning: integer constant is too large for 'long' typelib/percpu_test.c:89: warning: integer constant is too large for 'long' typelib/percpu_test.c:89: warning: integer constant is too large for 'long' typelib/percpu_test.c:89: warning: integer constant is too large for 'long' typelib/percpu_test.c:89: warning: integer constant is too large for 'long' typelib/percpu_test.c:97: warning: integer constant is too large for 'long' typelib/percpu_test.c:97: warning: integer constant is too large for 'long' typelib/percpu_test.c:97: warning: integer constant is too large for 'long' typelib/percpu_test.c:97: warning: integer constant is too large for 'long' type
Quoted 8 lines
lib/percpu_test.c:97: warning: integer constant is too large for 'long' typelib/percpu_test.c:97: warning: integer constant is too large for 'long' typelib/percpu_test.c:112: warning: integer constant is too large for 'long' typelib/percpu_test.c:112: warning: integer constant is too large for 'long' typelib/percpu_test.c:112: warning: integer constant is too large for 'long' typelib/percpu_test.c:112: warning: integer constant is too large for 'long' typelib/percpu_test.c:112: warning: integer constant is too large for 'long' typelib/percpu_test.c:112: warning: integer constant is too large for 'long' type
I was using gcc 4.6 which apparently adds LL suffix as needed. Though there were some other code problems with 32 bit beyond missing suffixes. Fixed version below tested with both gcc 4.4 and gcc 4.6 on 32 and 64 bit x86.
---8<---
From a95bb1ce42b4492644fa10c7c80fd9bbd7bf23b9 Mon Sep 17 00:00:00 2001
In-Reply-To: <20131104160918.0c571b410cf165e9c4b4a502-de/[email protected]>
References: <20131104160918.0c571b410cf165e9c4b4a502-de/[email protected]>
From: Greg Thelen <[email protected]>
Date: Sun, 27 Oct 2013 10:30:15 -0700
Subject: [PATCH v2] percpu: add test module for various percpu operations
Tests various percpu operations.
Enable with CONFIG_PERCPU_TEST=m.
---
Changelog since v1:
- use %lld/x which allows for less casting
- fix 32 bit build by casting large constants
lib/Kconfig.debug | 9 ++++ lib/Makefile | 2 + lib/percpu_test.c | 138 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 149 insertions(+) create mode 100644 lib/percpu_test.c
lib/Kconfig.debug +9 -0
@@ -1472,6 +1472,15 @@ config INTERVAL_TREE_TEST help A benchmark measuring the performance of the interval tree library +config PERCPU_TEST + tristate "Per cpu operations test" + depends on m && DEBUG_KERNEL + help + Enable this option to build test module which validates per-cpu + operations. + + If unsure, say N. + config ATOMIC64_SELFTEST bool "Perform an atomic64_t self-test at boot" help
lib/Makefile +2 -0
@@ -157,6 +157,8 @@ obj-$(CONFIG_INTERVAL_TREE_TEST) += interval_tree_test.o interval_tree_test-objs := interval_tree_test_main.o interval_tree.o +obj-$(CONFIG_PERCPU_TEST) += percpu_test.o + obj-$(CONFIG_ASN1) += asn1_decoder.o obj-$(CONFIG_FONT_SUPPORT) += fonts/
lib/percpu_test.c new file +138 -0
138 lines not on this page · raw message
Signature · 1 line
1.8.4.1
PATCH v2 2/3 percpu: fix this_cpu_sub() subtrahend casting for unsigneds
In reply to Greg Thelenthis_cpu_sub() is implemented as negation and addition.
This patch casts the adjustment to the counter type before negation to sign extend the adjustment. This helps in cases where the counter type is wider than an unsigned adjustment. An alternative to this patch is to declare such operations unsupported, but it seemed useful to avoid surprises.
This patch specifically helps the following example:
unsigned int delta = 1
preempt_disable() this_cpu_write(long_counter, 0) this_cpu_sub(long_counter, delta) preempt_enable()
Before this change long_counter on a 64 bit machine ends with value 0xffffffff, rather than 0xffffffffffffffff. This is because this_cpu_sub(pcp, delta) boils down to this_cpu_add(pcp, -delta), which is basically:
long_counter = 0 + 0xffffffff
Also apply the same cast to:
__this_cpu_sub()
__this_cpu_sub_return()
this_cpu_sub_return()
All percpu_test.ko passes, especially the following cases which previously failed:
l -= ui_one; __this_cpu_sub(long_counter, ui_one); CHECK(l, long_counter, -1); l -= ui_one; this_cpu_sub(long_counter, ui_one); CHECK(l, long_counter, -1); CHECK(l, long_counter, 0xffffffffffffffff); ul -= ui_one; __this_cpu_sub(ulong_counter, ui_one); CHECK(ul, ulong_counter, -1); CHECK(ul, ulong_counter, 0xffffffffffffffff); ul = this_cpu_sub_return(ulong_counter, ui_one); CHECK(ul, ulong_counter, 2); ul = __this_cpu_sub_return(ulong_counter, ui_one); CHECK(ul, ulong_counter, 1);
--- arch/x86/include/asm/percpu.h | 3 ++- include/linux/percpu.h | 8 ++++---- 2 files changed, 6 insertions(+), 5 deletions(-)
arch/x86/include/asm/percpu.h +2 -1
@@ -128,7 +128,8 @@ do { \ do { \ typedef typeof(var) pao_T__; \ const int pao_ID__ = (__builtin_constant_p(val) && \ - ((val) == 1 || (val) == -1)) ? (val) : 0; \ + ((val) == 1 || (val) == -1)) ? \ + (int)(val) : 0; \ if (0) { \ pao_T__ pao_tmp__; \ pao_tmp__ = (val); \
include/linux/percpu.h +4 -4
@@ -332,7 +332,7 @@ do { \#endif #ifndef this_cpu_sub -# define this_cpu_sub (pcp, val) this_cpu_add((pcp), -(val)) +# define this_cpu_sub (pcp, val) this_cpu_add((pcp), -(typeof(pcp))(val))#endif #ifndef this_cpu_inc
@@ -418,7 +418,7 @@ do { \# define this_cpu_add_return (pcp, val) __pcpu_size_call_return2(this_cpu_add_return_, pcp, val)#endif -#define this_cpu_sub_return (pcp, val) this_cpu_add_return(pcp, -(val)) +#define this_cpu_sub_return (pcp, val) this_cpu_add_return(pcp, -(typeof(pcp))(val))#define this_cpu_inc_return (pcp) this_cpu_add_return(pcp, 1)#define this_cpu_dec_return (pcp) this_cpu_add_return(pcp, -1)
@@ -586,7 +586,7 @@ do { \#endif #ifndef __this_cpu_sub -# define __this_cpu_sub (pcp, val) __this_cpu_add((pcp), -(val)) +# define __this_cpu_sub (pcp, val) __this_cpu_add((pcp), -(typeof(pcp))(val))#endif #ifndef __this_cpu_inc
@@ -668,7 +668,7 @@ do { \ __pcpu_size_call_return2(__this_cpu_add_return_, pcp, val)#endif -#define __this_cpu_sub_return (pcp, val) __this_cpu_add_return(pcp, -(val)) +#define __this_cpu_sub_return (pcp, val) __this_cpu_add_return(pcp, -(typeof(pcp))(val))#define __this_cpu_inc_return (pcp) __this_cpu_add_return(pcp, 1)#define __this_cpu_dec_return (pcp) __this_cpu_add_return(pcp, -1)
Signature · 7 lines
1.8.4.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH v2 2/3] percpu: fix this_cpu_sub() subtrahend casting for unsigneds
Quoted 49 lines · Greg Thelen
this_cpu_sub()is implemented as negation and addition.This patch casts the adjustment to the counter type before negation to sign extend the adjustment. This helps in cases where the counter type is wider than an unsigned adjustment. An alternative to this patch is to declare such operations unsupported, but it seemed useful to avoid surprises.
This patch specifically helps the following example:
unsigned int delta = 1
preempt_disable()this_cpu_write(long_counter, 0)this_cpu_sub(long_counter, delta)preempt_enable()Before this change
long_counteron a 64 bit machine ends with value0xffffffff, rather than0xffffffffffffffff. This is becausethis_cpu_sub(pcp, delta)boils down tothis_cpu_add(pcp, -delta), which is basically:
long_counter= 0 +0xffffffffAlso apply the same cast to:
__this_cpu_sub()
__this_cpu_sub_return()
this_cpu_sub_return()All percpu_test.ko passes, especially the following cases which previously failed:
l -= ui_one; __this_cpu_sub(long_counter, ui_one); CHECK(l, long_counter, -1); l -= ui_one; this_cpu_sub(long_counter, ui_one); CHECK(l, long_counter, -1); CHECK(l, long_counter, 0xffffffffffffffff); ul -= ui_one; __this_cpu_sub(ulong_counter, ui_one); CHECK(ul, ulong_counter, -1); CHECK(ul, ulong_counter, 0xffffffffffffffff); ul = this_cpu_sub_return(ulong_counter, ui_one); CHECK(ul, ulong_counter, 2); ul = __this_cpu_sub_return(ulong_counter, ui_one); CHECK(ul, ulong_counter, 1);
Quoted 2 lines
Signed-off-by Greg Thelen <[email protected]>Acked-by Tejun Heo <[email protected]>
FWIW:
PATCH v2 3/3 memcg: use __this_cpu_sub() to dec stats to avoid incorrect subtrahend casting
In reply to Greg ThelenAs of v3.11-9444-g3ea67d0 "memcg: add per cgroup writeback pages accounting" memcg counter errors are possible when moving charged memory to a different memcg. Charge movement occurs when processing writes to memory.force_empty, moving tasks to a memcg with memcg.move_charge_at_immigrate=1, or memcg deletion. An example showing error after memory.force_empty:
$ cd /sys/fs/cgroup/memory $ mkdir x $ rm /data/tmp/file $ (echo $BASHPID >> x/tasks && exec mmap_writer /data/tmp/file 1M) & [1] 13600 $ grep ^mapped x/memory.stat mapped_file 1048576 $ echo 13600 > tasks $ echo 1 > x/memory.force_empty $ grep ^mapped x/memory.stat mapped_file 4503599627370496
mapped_file should end with 0.
4503599627370496 == 0x10,0000,0000,0000 == 0x100,0000,0000 pages
1048576 == 0x10,0000 == 0x100 pages
This issue only affects the source memcg on 64 bit machines; the destination memcg counters are correct. So the rmdir case is not too important because such counters are soon disappearing with the entire memcg. But the memcg.force_empty and memory.move_charge_at_immigrate=1 cases are larger problems as the bogus counters are visible for the (possibly long) remaining life of the source memcg.
The problem is due to memcg use of __this_cpu_from(.., -nr_pages), which is subtly wrong because it subtracts the unsigned int nr_pages (either -1 or -512 for THP) from a signed long percpu counter. When nr_pages=-1, -nr_pages=0xffffffff. On 64 bit machines stat->count[idx] is signed 64 bit. So memcg's attempt to simply decrement a count (e.g. from 1 to 0) boils down to:
long count = 1
unsigned int nr_pages = 1
count += -nr_pages /* -nr_pages == 0xffff,ffff */ count is now 0x1,0000,0000 instead of 0
The fix is to subtract the unsigned page count rather than adding its negation. This only works once "percpu: fix this_cpu_sub() subtrahend casting for unsigneds" is applied to fix this_cpu_sub().
--- mm/memcontrol.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/memcontrol.c +1 -1
@@ -3773,7 +3773,7 @@ void mem_cgroup_move_account_page_stat(struct mem_cgroup *from, {/* Update stat data for mem_cgroup */ preempt_disable (); -__this_cpu_add (from->stat->count[idx], -nr_pages); +__this_cpu_sub (from->stat->count[idx], nr_pages);__this_cpu_add (to->stat->count[idx], nr_pages);preempt_enable (); }
Signature · 7 lines
1.8.4.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH v2 3/3] memcg: use __this_cpu_sub() to dec stats to avoid incorrect subtrahend casting
Quoted 41 lines · Greg Thelen
As of v3.11-9444-g3ea67d0 "memcg: add per cgroup writeback pages accounting" memcg counter errors are possible when moving charged memory to a different memcg. Charge movement occurs when processing writes to memory.force_empty, moving tasks to a memcg with memcg.move_charge_at_immigrate=1, or memcg deletion. An example showing error after memory.force_empty:
$ cd /sys/fs/cgroup/memory $ mkdir x $ rm /data/tmp/file $ (echo $BASHPID >> x/tasks && exec mmap_writer /data/tmp/file 1M) & [1] 13600 $ grep ^mapped x/memory.stat mapped_file 1048576 $ echo 13600 > tasks $ echo 1 > x/memory.force_empty $ grep ^mapped x/memory.stat mapped_file 4503599627370496
mapped_fileshould end with 0.
4503599627370496 == 0x10,0000,0000,0000 == 0x100,0000,0000 pages
1048576 == 0x10,0000 ==0x100pagesThis issue only affects the source memcg on 64 bit machines; the destination memcg counters are correct. So the rmdir case is not too important because such counters are soon disappearing with the entire memcg. But the memcg.force_empty and memory.move_charge_at_immigrate=1 cases are larger problems as the bogus counters are visible for the (possibly long) remaining life of the source memcg.
The problem is due to memcg use of
__this_cpu_from(.., -nr_pages), which is subtly wrong because it subtracts the unsigned intnr_pages(either -1 or -512 for THP) from a signed long percpu counter. When nr_pages=-1, -nr_pages=0xffffffff. On 64 bit machines stat->count[idx] is signed 64 bit. So memcg's attempt to simply decrement a count (e.g. from 1 to 0) boils down to:
long count = 1
unsigned intnr_pages= 1
count += -nr_pages /* -nr_pages == 0xffff,ffff */ count is now 0x1,0000,0000 instead of 0
Quoted 6 lines
The fix is to subtract the unsigned page count rather than adding its negation. This only works once "percpu: fix
this_cpu_sub()subtrahend casting for unsigneds" is applied to fixthis_cpu_sub().Signed-off-by Greg Thelen <[email protected]>Acked-by Tejun Heo <[email protected]>
Huh, it looked so innocent... At first I thought 2/3 would fix this case as well but the cast happens only after the negation, so the sign extension does not happen. Alright, then.