amusing SLUB compaction bug when CC_OPTIMIZE_FOR_SIZE
Subsystems (4)
It's a bug in linux-next, but taking me too long to identify which commit is "to blame", so let me throw it over to you without more
delay: I think __PageMovable() now needs to check !PageSlab().
I had made a small experimental change somewhere, rebuilt and rebooted, was not surprised to crash once swapping and compaction came in, but was surprised to find the crash in isolate_movable_page(), called by compaction's isolate_migratepages_block().
page->mapping was ffffffff811303aa, which qualifies as __PageMovable(), which expects struct movable_operations at page->mapping minus low bits. But ffffffff811303aa was the address of SLUB's rcu_free_slab(): I have CONFIG_CC_OPTIMIZE_FOR_SIZE=y, so function addresses may have low bits set.
Over to you! Thanks,
Hugh
Quoted 9 lines · Hugh Dickins
It's a bug in linux-next, but taking me too long to identify which commit is "to blame", so let me throw it over to you without more
delay: I think__PageMovable()now needs to check !PageSlab().I had made a small experimental change somewhere, rebuilt and rebooted, was not surprised to crash once swapping and compaction came in, but was surprised to find the crash in
isolate_movable_page(), called by compaction'sisolate_migratepages_block().
Quoted 7 lines
page->mappingwasffffffff811303aa, which qualifies as__PageMovable(), which expects structmovable_operationsatpage->mappingminus low bits. Butffffffff811303aawas the address of SLUB'srcu_free_slab(): I haveCONFIG_CC_OPTIMIZE_FOR_SIZE=y, so function addresses may have low bits set.Over to you! Thanks,
Hugh
Wow, didn't expect this.
Thank you for report!
That should be due to commit 65505d1f2338e7
("mm/sl[au]b: rearrange struct slab fields to allow larger rcu_head") as now rcu_head can use some bits that shares with mapping.
Hmm IMO we have two choices...
1. simply drop the commit as it's only for debugging (RCU folks may not like [1])
2. make __PageMovable() to use true page flag, with approach [2])
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
Thanks!
Signature · 2 lines
Thanks, Hyeonggon
Quoted 21 lines · Hyeonggon Yoo
Quoted 9 lines · Hugh Dickins
It's a bug in linux-next, but taking me too long to identify which commit is "to blame", so let me throw it over to you without more
delay: I think__PageMovable()now needs to check !PageSlab().I had made a small experimental change somewhere, rebuilt and rebooted, was not surprised to crash once swapping and compaction came in, but was surprised to find the crash in
isolate_movable_page(), called by compaction'sisolate_migratepages_block().Quoted 7 lines
page->mappingwasffffffff811303aa, which qualifies as__PageMovable(), which expects structmovable_operationsatpage->mappingminus low bits. Butffffffff811303aawas the address of SLUB'srcu_free_slab(): I haveCONFIG_CC_OPTIMIZE_FOR_SIZE=y, so function addresses may have low bits set.Over to you! Thanks,
HughWow, didn't expect this.
Thank you for report!
Quoted 7 lines
That should be due to commit
65505d1f2338e7
("mm/sl[au]b: rearrange struct slab fields to allow largerrcu_head") as nowrcu_headcan use some bits that shares with mapping.Hmm IMO we have two choices...
1. simply drop the commit as it's only for debugging (RCU folks may not like [1])
Yeah definitely don't like this option as patches are out that depend on this (not yet merged though). :-)
Quoted 1 line
2. make
__PageMovable()to use true page flag, with approach [2])
What are the drawbacks of making it a true flag?
thanks,
- Joel
Quoted 8 lines
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/Thanks!
Signature · 2 lines
Thanks, Hyeonggon
Quoted 26 lines · Joel Fernandes
Quoted 25 lines · Hyeonggon Yoo
Quoted 9 lines · Hugh Dickins
It's a bug in linux-next, but taking me too long to identify which commit is "to blame", so let me throw it over to you without more
delay: I think__PageMovable()now needs to check !PageSlab().I had made a small experimental change somewhere, rebuilt and rebooted, was not surprised to crash once swapping and compaction came in, but was surprised to find the crash in
isolate_movable_page(), called by compaction'sisolate_migratepages_block().Quoted 7 lines
page->mappingwasffffffff811303aa, which qualifies as__PageMovable(), which expects structmovable_operationsatpage->mappingminus low bits. Butffffffff811303aawas the address of SLUB'srcu_free_slab(): I haveCONFIG_CC_OPTIMIZE_FOR_SIZE=y, so function addresses may have low bits set.Over to you! Thanks,
HughWow, didn't expect this.
Thank you for report!That should be due to commit
65505d1f2338e7
("mm/sl[au]b: rearrange struct slab fields to allow largerrcu_head") as nowrcu_headcan use some bits that shares with mapping.
Quoted 6 lines
Quoted 3 lines
Hmm IMO we have two choices...
1. simply drop the commit as it's only for debugging (RCU folks may not like [1])
Yeah definitely don't like this option as patches are out that depend on this (not yet merged though). :-)
Yeah, better to avoid that.
Quoted 3 lines
Quoted 1 line
2. make
__PageMovable()to use true page flag, with approach [2])What are the drawbacks of making it a true flag?
We don't have free bit in page flags unless we free one ;)
Quoted 5 lines
thanks,
- Joel
Signature · 2 lines
Thanks, Hyeonggon
Quoted 5 lines · Joel Fernandes
Quoted 4 lines · Hyeonggon Yoo
Quoted 3 lines · Hugh Dickins
It's a bug in linux-next, but taking me too long to identify which commit is "to blame", so let me throw it over to you without more
delay: I think__PageMovable()now needs to check !PageSlab().
When I tried that, the result wasn't really nice:
https://lore.kernel.org/all/[email protected]/
And what if there's another conflicting page "type" later. Or the debugging variant of rcu_head in struct page itself. The __PageMovable() is just too fragile.
Quoted 20 lines
Quoted 20 lines
Quoted 5 lines
I had made a small experimental change somewhere, rebuilt and rebooted, was not surprised to crash once swapping and compaction came in, but was surprised to find the crash in
isolate_movable_page(), called by compaction'sisolate_migratepages_block().Quoted 7 lines
page->mappingwasffffffff811303aa, which qualifies as__PageMovable(), which expects structmovable_operationsatpage->mappingminus low bits. Butffffffff811303aawas the address of SLUB'srcu_free_slab(): I haveCONFIG_CC_OPTIMIZE_FOR_SIZE=y, so function addresses may have low bits set.Over to you! Thanks,
HughWow, didn't expect this.
Thank you for report!That should be due to commit
65505d1f2338e7
("mm/sl[au]b: rearrange struct slab fields to allow largerrcu_head") as nowrcu_headcan use some bits that shares with mapping.
Quoted 6 lines
Quoted 3 lines
Hmm IMO we have two choices...
1. simply drop the commit as it's only for debugging (RCU folks may not like [1])
Yeah definitely don't like this option as patches are out that depend on this (not yet merged though). :-)
But we'll have to do that for now and postpone to 6.2 I'm afraid as merge window for 6.1 is too close to have confidence in any solution that we came up this moment.
Quoted 3 lines
Quoted 1 line
2. make
__PageMovable()to use true page flag, with approach [2])What are the drawbacks of making it a true flag?
Even if we free PageSlab, I'm sure there will be better uses of a free page flag than __PageMovable.
3. With frozen page allocation
https://lore.kernel.org/all/[email protected]/
slab pages will have refcount 0 and compaction will skip them for that reason. But it had other unresolved issues with page isolation code IIRC.
Quoted 15 lines
thanks,
- Joel
Quoted 8 lines
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/Thanks!
Signature · 2 lines
Thanks, Hyeonggon
Quoted 7 lines · Vlastimil Babka
Quoted 5 lines · Joel Fernandes
Quoted 4 lines · Hyeonggon Yoo
Quoted 3 lines · Hugh Dickins
It's a bug in linux-next, but taking me too long to identify which commit is "to blame", so let me throw it over to you without more
delay: I think__PageMovable()now needs to check !PageSlab().
Quoted 7 lines
When I tried that, the result wasn't really nice:
https://lore.kernel.org/all/[email protected]/
And what if there's another conflicting page "type" later. Or the debugging variant of
rcu_headin struct page itself. The__PageMovable()is just too fragile.
I don't disagree (and don't really know all the things you're thinking of in there). But if it's important to rescue this feature for 6.1, a different approach may be the very simple patch below (I met a similar issue with OPTIMIZE_FOR_SIZE in i915 a year ago, and just remembered).
But you be the judge of it: (a) I do not know whether rcu_free_slab is the only risky address ever stuffed into that field; and (b) I'm clueless when it comes to those architectures (powerpc etc) where the the address of a function is something different from the address of the function (have I conveyed my cluelessness adequately?).
Hugh
mm/slub.c +6 -1
@@ -1953,7 +1953,12 @@ static void __free_slab(struct kmem_cach__free_pages (folio_page (folio,0 ), order); } -static void rcu_free_slab (struct rcu_head *h) +/* +* rcu_free_slab() must be __aligned(4) because its address is saved +* in the rcu_head field, which coincides with page->mapping, which +* causes trouble if compaction mistakes it for PAGE_MAPPING_MOVABLE. +*/ +__aligned(4) static void rcu_free_slab (struct rcu_head *h) { struct slab *slab = container_of(h, struct slab, rcu_head);
Quoted 37 lines · Vlastimil Babka
Quoted 5 lines · Joel Fernandes
Quoted 4 lines · Hyeonggon Yoo
Quoted 3 lines · Hugh Dickins
It's a bug in linux-next, but taking me too long to identify which commit is "to blame", so let me throw it over to you without more
delay: I think__PageMovable()now needs to check !PageSlab().When I tried that, the result wasn't really nice:
https://lore.kernel.org/all/[email protected]/
And what if there's another conflicting page "type" later. Or the debugging variant of
rcu_headin struct page itself. The__PageMovable()is just too fragile.Quoted 22 lines
Quoted 22 lines
Quoted 5 lines
I had made a small experimental change somewhere, rebuilt and rebooted, was not surprised to crash once swapping and compaction came in, but was surprised to find the crash in
isolate_movable_page(), called by compaction'sisolate_migratepages_block().Quoted 7 lines
page->mappingwasffffffff811303aa, which qualifies as__PageMovable(), which expects structmovable_operationsatpage->mappingminus low bits. Butffffffff811303aawas the address of SLUB'srcu_free_slab(): I haveCONFIG_CC_OPTIMIZE_FOR_SIZE=y, so function addresses may have low bits set.Over to you! Thanks,
HughWow, didn't expect this.
Thank you for report!That should be due to commit
65505d1f2338e7
("mm/sl[au]b: rearrange struct slab fields to allow largerrcu_head") as nowrcu_headcan use some bits that shares with mapping.Hmm IMO we have two choices...
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
1. simply drop the commit as it's only for debugging (RCU folks may not like [1])
Yeah definitely don't like this option as patches are out that depend on this (not yet merged though). :-)
But we'll have to do that for now and postpone to 6.2 I'm afraid as merge window for 6.1 is too close to have confidence in any solution that we came up this moment.
Agreed.
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
2. make
__PageMovable()to use true page flag, with approach [2])What are the drawbacks of making it a true flag?
Quoted 9 lines
Even if we free PageSlab, I'm sure there will be better uses of a free page flag than
__PageMovable.3. With frozen page allocation
https://lore.kernel.org/all/[email protected]/slab pages will have refcount 0 and compaction will skip them for that reason. But it had other unresolved issues with page isolation code IIRC.
4. Always align function rcu_free_slab() with 'aligned' attribute
[1] https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-aligned-function-attribute
[2] __aligned in include/linux/compiler_attributes.h
Quoted 16 lines
Quoted 15 lines
thanks,
- Joel
Quoted 8 lines
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/Thanks!
Signature · 2 lines
Thanks, Hyeonggon
Signature · 2 lines
Thanks, Hyeonggon
Quoted 4 lines · Hyeonggon Yoo
On Wed, Sep 28, 2022 at 06:20:10PM +0200, Vlastimil Babka wrote:
[..]
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Thank you for report!
That should be due to commit
65505d1f2338e7
("mm/sl[au]b: rearrange struct slab fields to allow largerrcu_head") as nowrcu_headcan use some bits that shares with mapping.Hmm IMO we have two choices...
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
1. simply drop the commit as it's only for debugging (RCU folks may not like [1])
Yeah definitely don't like this option as patches are out that depend on this (not yet merged though). :-)
But we'll have to do that for now and postpone to 6.2 I'm afraid as merge window for 6.1 is too close to have confidence in any solution that we came up this moment.
I am ok with your postponing till then, gives me time to get other patches in, in the mean while :-)
Thanks again for your work on this,
- Joel
Quoted 21 lines · Hugh Dickins
Quoted 7 lines · Vlastimil Babka
Quoted 5 lines · Joel Fernandes
Quoted 4 lines · Hyeonggon Yoo
Quoted 3 lines · Hugh Dickins
It's a bug in linux-next, but taking me too long to identify which commit is "to blame", so let me throw it over to you without more
delay: I think__PageMovable()now needs to check !PageSlab().Quoted 7 lines
When I tried that, the result wasn't really nice:
https://lore.kernel.org/all/[email protected]/
And what if there's another conflicting page "type" later. Or the debugging variant of
rcu_headin struct page itself. The__PageMovable()is just too fragile.I don't disagree (and don't really know all the things you're thinking of in there). But if it's important to rescue this feature for 6.1, a different approach may be the very simple patch below (I met a similar issue with
OPTIMIZE_FOR_SIZEin i915 a year ago, and just remembered).
Quoted 5 lines
But you be the judge of it: (a) I do not know whether
rcu_free_slabis the only risky address ever stuffed into that field; and (b) I'm clueless when it comes to those architectures (powerpc etc) where the the address of a function is something different from the address of the function (have I conveyed my cluelessness adequately?).
Thanks a lot Hugh! That's a sufficiently small fix (compared to the other options) that I'm probably give it one last try.
Quoted 18 lines
Hugh
mm/slub.c +6
-1@@ -1953,7 +1953,12 @@ static void __free_slab(struct kmem_cach__free_pages (folio_page (folio,0 ), order); } -static void rcu_free_slab (struct rcu_head *h) +/* +* rcu_free_slab() must be __aligned(4) because its address is saved +* in the rcu_head field, which coincides with page->mapping, which +* causes trouble if compaction mistakes it for PAGE_MAPPING_MOVABLE. +*/ +__aligned(4) static void rcu_free_slab (struct rcu_head *h) { struct slab *slab = container_of(h, struct slab, rcu_head);
mm/slub.c static void __free_slab(struct kmem_cach · line 1963
Quoted 10 lines
mm/slub.c +6
-1-static void rcu_free_slab (struct rcu_head *h) +/* +* rcu_free_slab() must be __aligned(4) because its address is saved +* in the rcu_head field, which coincides with page->mapping, which +* causes trouble if compaction mistakes it for PAGE_MAPPING_MOVABLE. +*/ +__aligned(4) static void rcu_free_slab (struct rcu_head *h) { struct slab *slab = container_of(h, struct slab, rcu_head);
Isn't that going to cause grief with options that align functions on 16/32byte boundaries when adding space for 'other stuff'?
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 6 lines · David Laight
Quoted 6 lines
mm/slub.c +5
-1-static void rcu_free_slab (struct rcu_head *h) +/* +* rcu_free_slab() must be __aligned(4) because its address is saved +* in the rcu_head field, which coincides with page->mapping, which +* causes trouble if compaction mistakes it for PAGE_MAPPING_MOVABLE. +*/
Quoted 8 lines
mm/slub.c static void __free_slab(struct kmem_cach · line 1963
Quoted 4 lines
mm/slub.c +1
-0+__aligned(4) static void rcu_free_slab (struct rcu_head *h) { struct slab *slab = container_of(h, struct slab, rcu_head);Isn't that going to cause grief with options that align functions on 16/32byte boundaries when adding space for 'other stuff'?
How is that done exactly? Also having higher alignment (16/32) is not in conflict with asking for 4?
Quoted 6 lines
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
From: Vlastimil Babka
Quoted 18 lines
Sent: 29 September 2022 14:01
Quoted 6 lines · David Laight
Quoted 6 lines
mm/slub.c +5
-1-static void rcu_free_slab (struct rcu_head *h) +/* +* rcu_free_slab() must be __aligned(4) because its address is saved +* in the rcu_head field, which coincides with page->mapping, which +* causes trouble if compaction mistakes it for PAGE_MAPPING_MOVABLE. +*/ Quoted 8 lines
mm/slub.c static void __free_slab(struct kmem_cach · line 1963
Quoted 4 lines
mm/slub.c +1
-0+__aligned(4) static void rcu_free_slab (struct rcu_head *h) { struct slab *slab = container_of(h, struct slab, rcu_head);Isn't that going to cause grief with options that align functions on 16/32byte boundaries when adding space for 'other stuff'?
Quoted 2 lines
How is that done exactly? Also having higher alignment (16/32) is not in conflict with asking for 4?
It depends on which one is actually used.
The __aligned(4) might take precedence over the default alignment.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 22 lines · Vlastimil Babka
Quoted 21 lines · Hugh Dickins
Quoted 7 lines · Vlastimil Babka
Quoted 5 lines · Joel Fernandes
Quoted 4 lines · Hyeonggon Yoo
Quoted 3 lines · Hugh Dickins
It's a bug in linux-next, but taking me too long to identify which commit is "to blame", so let me throw it over to you without more
delay: I think__PageMovable()now needs to check !PageSlab().Quoted 7 lines
When I tried that, the result wasn't really nice:
https://lore.kernel.org/all/[email protected]/
And what if there's another conflicting page "type" later. Or the debugging variant of
rcu_headin struct page itself. The__PageMovable()is just too fragile.I don't disagree (and don't really know all the things you're thinking of in there). But if it's important to rescue this feature for 6.1, a different approach may be the very simple patch below (I met a similar issue with
OPTIMIZE_FOR_SIZEin i915 a year ago, and just remembered).
Quoted 8 lines
Quoted 5 lines
But you be the judge of it: (a) I do not know whether
rcu_free_slabis the only risky address ever stuffed into that field; and (b) I'm clueless when it comes to those architectures (powerpc etc) where the the address of a function is something different from the address of the function (have I conveyed my cluelessness adequately?).Thanks a lot Hugh! That's a sufficiently small fix (compared to the other options) that I'm probably give it one last try.
I suddenly worried that you might be waiting on me for a Signed-off-by, which I couldn't give until I researched my reservations (a) and (b):
but I'm pleased to see from your kernel.org tree that you've gone ahead and folded it in - thanks.
Regarding (a): great, you've found it too, mm/slab.c's kmem_rcu_free() looks like it needs the same __aligned(4) as mm/slub.c's rcu_free_slabi().
Regarding (b): I booted the PowerMac G5 to take a look, and dredged up the relevant phrase "function descriptor" from depths of my memory: I was right to consider that case, but it's not a worry - the first field of a function descriptor structure (on all the architectures I found it) is the function address, so the function descriptor address would be aligned 4 or 8 anyway.
Regarding "conflicting" alignment requests: yes, I agree with you, it would have to be a toolchain bug if when asked to align 2 and to align 4, it chose not to align 4.
So, no worries at my end now.
Hugh
Quoted 29 lines · Hugh Dickins
Quoted 28 lines · Vlastimil Babka
Quoted 26 lines · Hugh Dickins
Quoted 7 lines · Vlastimil Babka
Quoted 5 lines · Joel Fernandes
Quoted 4 lines · Hyeonggon Yoo
Quoted 3 lines · Hugh Dickins
It's a bug in linux-next, but taking me too long to identify which commit is "to blame", so let me throw it over to you without more
delay: I think__PageMovable()now needs to check !PageSlab().Quoted 7 lines
When I tried that, the result wasn't really nice:
https://lore.kernel.org/all/[email protected]/
And what if there's another conflicting page "type" later. Or the debugging variant of
rcu_headin struct page itself. The__PageMovable()is just too fragile.I don't disagree (and don't really know all the things you're thinking of in there). But if it's important to rescue this feature for 6.1, a different approach may be the very simple patch below (I met a similar issue with
OPTIMIZE_FOR_SIZEin i915 a year ago, and just remembered).But you be the judge of it: (a) I do not know whether
rcu_free_slabis the only risky address ever stuffed into that field; and (b) I'm clueless when it comes to those architectures (powerpc etc) where the the address of a function is something different from the address of the function (have I conveyed my cluelessness adequately?).
Quoted 7 lines
Quoted 2 lines
Thanks a lot Hugh! That's a sufficiently small fix (compared to the other options) that I'm probably give it one last try.
I suddenly worried that you might be waiting on me for a Signed-off-by, which I couldn't give until I researched my reservations (a) and (b):
but I'm pleased to see from your kernel.org tree that you've gone ahead and folded it in - thanks.
Yeah could have been more explicit about that, sorry. But made the whole thing a very last merge so I can still drop it before the pull request.
Quoted 2 lines
Regarding (a): great, you've found it too, mm/slab.c's
kmem_rcu_free()looks like it needs the same__aligned(4)as mm/slub.c'srcu_free_slabi().
Right.
Quoted 6 lines
Regarding (b): I booted the PowerMac G5 to take a look, and dredged up the relevant phrase "function descriptor" from depths of my memory: I was right to consider that case, but it's not a worry - the first field of a function descriptor structure (on all the architectures I found it) is the function address, so the function descriptor address would be aligned 4 or 8 anyway.
Thanks. I admit I wasn't that thorough, just consulted somebody internally :)
Quoted 3 lines
Regarding "conflicting" alignment requests: yes, I agree with you, it would have to be a toolchain bug if when asked to align 2 and to align 4, it chose not to align 4.
Yeah. But I still would be less worried if another __aligned(X) function existed in the tree already. Found only data. I assume the i915 thing wasn't fixed like this in the tree? So if there are buggy toolchains or anything, it will be us to discover them.
So I think we still should defuse the __PageMovable() mine somehow.
Quoted 2 lines
So, no worries at my end now.
Hugh
Quoted 6 lines · Vlastimil Babka
Quoted 5 lines · Hugh Dickins
Quoted 3 lines · Vlastimil Babka
Thanks a lot Hugh! That's a sufficiently small fix (compared to the other options) that I'm probably give it one last try.
Quoted 7 lines
Quoted 4 lines
I suddenly worried that you might be waiting on me for a Signed-off-by, which I couldn't give until I researched my reservations (a) and (b):
but I'm pleased to see from your kernel.org tree that you've gone ahead and folded it in - thanks.Yeah could have been more explicit about that, sorry. But made the whole thing a very last merge so I can still drop it before the pull request.
No probs, you did the right thing.
Quoted 6 lines
Quoted 3 lines
Regarding (a): great, you've found it too, mm/slab.c's
kmem_rcu_free()looks like it needs the same__aligned(4)as mm/slub.c'srcu_free_slabi().Right.
Quoted 8 lines
Quoted 6 lines
Regarding (b): I booted the PowerMac G5 to take a look, and dredged up the relevant phrase "function descriptor" from depths of my memory: I was right to consider that case, but it's not a worry - the first field of a function descriptor structure (on all the architectures I found it) is the function address, so the function descriptor address would be aligned 4 or 8 anyway.
Thanks. I admit I wasn't that thorough, just consulted somebody internally :)
Exactly what I had hoped you would do.
Quoted 9 lines
Quoted 4 lines
Regarding "conflicting" alignment requests: yes, I agree with you, it would have to be a toolchain bug if when asked to align 2 and to align 4, it chose not to align 4.
Yeah. But I still would be less worried if another
__aligned(X)function existed in the tree already. Found only data. I assume the i915 thing wasn't fixed like this in the tree? So if there are buggy toolchains or anything, it will be us to discover them.
Linus put it in himself, after it had got lost over several -rcs:
5.15's cdc1e6e225e3 ("drm/i915: fix blank screen booting crashes").
Originally I'd written "__aligned(4)" explicitly, but later found i915 relied on it elsewhere since 4.9, and had an __i915_sw_fence_call for it.
But 5.17's 44505168d743 ("drm/i915: Drop stealing of bits from i915_sw_fence function pointer") appears to have removed all that now.
I think that gives assurance that the x86 toolchains are okay; but I imagine i915 is unlikely to be found on other architectures, so not quite so much assurance there.
Hugh
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Regarding "conflicting" alignment requests: yes, I agree with you, it would have to be a toolchain bug if when asked to align 2 and to align 4, it chose not to align 4.
See https://godbolt.org/z/3nGsTaf5e
the align() directive takes precedence.
Here you only want to ensure the alignment is at least 4.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 8 lines · David Laight
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Regarding "conflicting" alignment requests: yes, I agree with you, it would have to be a toolchain bug if when asked to align 2 and to align 4, it chose not to align 4.
See https://godbolt.org/z/3nGsTaf5e
thealign()directive takes precedence.Here you only want to ensure the alignment is at least 4.
Sorry, I don't understand the point you are making, nor how to deduce it from the link which you give:
I'll leave it to those who understand better.
Hugh
From: Hugh Dickins
Quoted 7 lines
Sent: 30 September 2022 17:22
Quoted 4 lines · David Laight
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Regarding "conflicting" alignment requests: yes, I agree with you, it would have to be a toolchain bug if when asked to align 2 and to align 4, it chose not to align 4.
Quoted 8 lines
Quoted 4 lines
See https://godbolt.org/z/3nGsTaf5e
thealign()directive takes precedence.Here you only want to ensure the alignment is at least 4.
Sorry, I don't understand the point you are making, nor how to deduce it from the link which you give:
I'll leave it to those who understand better.
(I've copied Kees Cook - I think it is his patches that are increasing the function alignment.
He'll need to find the thread history...)
IIRC -malign_functions is being used to put functions onto 16 byte boundaries to get aligned pad space before the function. Adding __align(4) overrides this and (I think) will remove the pad space.
I could only see the alignment directives in gcc ouput. clang didn't seem to generate them.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 32 lines · Hugh Dickins
Quoted 22 lines · Vlastimil Babka
Quoted 21 lines · Hugh Dickins
Quoted 7 lines · Vlastimil Babka
Quoted 5 lines · Joel Fernandes
Quoted 4 lines · Hyeonggon Yoo
Quoted 3 lines · Hugh Dickins
It's a bug in linux-next, but taking me too long to identify which commit is "to blame", so let me throw it over to you without more
delay: I think__PageMovable()now needs to check !PageSlab().Quoted 7 lines
When I tried that, the result wasn't really nice:
https://lore.kernel.org/all/[email protected]/
And what if there's another conflicting page "type" later. Or the debugging variant of
rcu_headin struct page itself. The__PageMovable()is just too fragile.I don't disagree (and don't really know all the things you're thinking of in there). But if it's important to rescue this feature for 6.1, a different approach may be the very simple patch below (I met a similar issue with
OPTIMIZE_FOR_SIZEin i915 a year ago, and just remembered).Quoted 8 lines
Quoted 5 lines
But you be the judge of it: (a) I do not know whether
rcu_free_slabis the only risky address ever stuffed into that field; and (b) I'm clueless when it comes to those architectures (powerpc etc) where the the address of a function is something different from the address of the function (have I conveyed my cluelessness adequately?).Thanks a lot Hugh! That's a sufficiently small fix (compared to the other options) that I'm probably give it one last try.
Quoted 7 lines
I suddenly worried that you might be waiting on me for a Signed-off-by, which I couldn't give until I researched my reservations (a) and (b):
but I'm pleased to see from your kernel.org tree that you've gone ahead and folded it in - thanks.Regarding (a): great, you've found it too, mm/slab.c's
kmem_rcu_free()looks like it needs the same__aligned(4)as mm/slub.c'srcu_free_slabi().
Just one more thing, rcu_leak_callback too. RCU seem to use it internally to catch double call_rcu().
And some suggestions:
- what about adding runtime WARN() on slab init code to catch unexpected arch/toolchain issues?
- instead of 4, we may use macro definition? like (PAGE_MAPPING_FLAGS + 1)?
Signature · 2 lines
Thanks, Hyeonggon
Quoted 7 lines · Hyeonggon Yoo
Just one more thing,
rcu_leak_callbacktoo. RCU seem to use it internally to catch doublecall_rcu().And some suggestions:
- what about adding runtimeWARN()on slab init code to catch unexpectedarch/toolchainissues?
- instead of 4, we may use macro definition? like (PAGE_MAPPING_FLAGS+ 1)?
I think the real problem here is that isolate_movable_page() is insufficiently paranoid. Looking at the gyrations that GUP and the page cache do to convince themselves that the page they got really is the page they wanted, there are a few missing pieces (eg checking that you actually got a refcount on _this_ page and not some random other page you were temporarily part of a compound page with).
This patch does three things:
- Turns one of the comments into English. There are some others which I'm still scratching my head over.
- Uses a folio to help distinguish which operations are being done to the head vs the specific page (this is somewhat an abuse of the folio concept, but it's acceptable)
- Add the aforementioned check that we're actually operating on the page that we think we want to be.
- Add a check that the folio isn't secretly a slab.
We could put the slab check in PageMapping and call it after taking the folio lock, but that seems pointless. It's the acquisition of the refcount which stabilises the slab flag, not holding the lock.
mm/migrate.c +20 -12
@@ -59,6 +59,7 @@int isolate_movable_page (struct page *page,isolate_mode_t mode) { +struct folio *folio =page_folio (page);const struct movable_operations *mops;/*
@@ -70,16 +71,23 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)* the put_page() at the end of this block will take care of * release this page, thus avoiding a nasty leakage. */ -if (unlikely (!get_page_unless_zero (page))) +if (unlikely (!folio_try_get (folio))) goto out; +/* Recheck the page is still part of the folio we just got */ +if (unlikely (page_folio (page) != folio)) + goto out_put; +/* -* Check PageMovable before holding a PG_lock because page's owner -* assumes anybody doesn't touch PG_lock of newly allocated page -* so unconditionally grabbing the lock ruins page's owner side. +* Check movable flag before taking the folio lock because +* we use non-atomic bitops on newly allocated page flags so +* unconditionally grabbing the lock ruins page's owner side. */ -if (unlikely (!__PageMovable (page))) - goto out_putpage; +if (unlikely (!__folio_test_movable (folio))) + goto out_put; +if (unlikely (folio_test_slab (folio))) + goto out_put; +/* * As movable pages are not isolated from LRU lists, concurrent * compaction threads can race against page migration functions
@@ -91,8 +99,8 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)* lets be sure we have the page lock * before proceeding with the movable page isolation steps. */ -if (unlikely (!trylock_page (page))) - goto out_putpage; +if (unlikely (!folio_trylock (folio))) + goto out_put;if (!PageMovable (page) ||PageIsolated (page)) goto out_no_isolated;
@@ -106,14 +114,14 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)/* Driver shouldn't use PG_isolated bit of page->flags */ WARN_ON_ONCE (PageIsolated (page));SetPageIsolated (page); -unlock_page (page); +folio_unlock (folio);return 0 ; out_no_isolated: -unlock_page (page); -out_putpage: -put_page (page); +folio_unlock (folio); +out_put: +folio_put (folio); out:return -EBUSY ; }
Quoted 27 lines · Matthew Wilcox
Quoted 7 lines · Hyeonggon Yoo
Just one more thing,
rcu_leak_callbacktoo. RCU seem to use it internally to catch doublecall_rcu().And some suggestions:
- what about adding runtimeWARN()on slab init code to catch unexpectedarch/toolchainissues?
- instead of 4, we may use macro definition? like (PAGE_MAPPING_FLAGS+ 1)?I think the real problem here is that
isolate_movable_page()is insufficiently paranoid. Looking at the gyrations that GUP and the page cache do to convince themselves that the page they got really is the page they wanted, there are a few missing pieces (eg checking that you actually got a refcount on _this_ page and not some random other page you were temporarily part of a compound page with).This patch does three things:
- Turns one of the comments into English. There are some others which I'm still scratching my head over.
- Uses a folio to help distinguish which operations are being done to the head vs the specific page (this is somewhat an abuse of the folio concept, but it's acceptable)
- Add the aforementioned check that we're actually operating on the page that we think we want to be.
- Add a check that the folio isn't secretly a slab.
Quoted 2 lines
We could put the slab check in PageMapping and call it after taking the folio lock, but that seems pointless
I partially agree with this patch. I actually like it.
Quoted 2 lines
It's the acquisition of
the refcount which stabilises the slab flag, not holding the lock.
But can you please elaborate how this prevents race between allocation & initialization of a slab and isolate_movable_page()?
Or maybe we can handle it with frozen folio as Vlastimil suggested? ;-)
Quoted 43 lines
mm/migrate.c +14
-6@@ -59,6 +59,7 @@int isolate_movable_page (struct page *page,isolate_mode_t mode) { +struct folio *folio =page_folio (page);const struct movable_operations *mops;/* @@ -70,16 +71,23 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)* the put_page() at the end of this block will take care of * release this page, thus avoiding a nasty leakage. */ -if (unlikely (!get_page_unless_zero (page))) +if (unlikely (!folio_try_get (folio))) goto out; +/* Recheck the page is still part of the folio we just got */ +if (unlikely (page_folio (page) != folio)) + goto out_put; +/* -* Check PageMovable before holding a PG_lock because page's owner -* assumes anybody doesn't touch PG_lock of newly allocated page -* so unconditionally grabbing the lock ruins page's owner side. +* Check movable flag before taking the folio lock because +* we use non-atomic bitops on newly allocated page flags so +* unconditionally grabbing the lock ruins page's owner side. */ -if (unlikely (!__PageMovable (page))) - goto out_putpage; +if (unlikely (!__folio_test_movable (folio))) + goto out_put; +if (unlikely (folio_test_slab (folio))) + goto out_put; +/* * As movable pages are not isolated from LRU lists, concurrent * compaction threads can race against page migration functions
mm/migrate.c int isolate_movable_page(struct page *page, isolate_mode_t mode) · line 103
Quoted 7 lines
mm/migrate.c +2
-2@@ -91,8 +99,8 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)* lets be sure we have the page lock * before proceeding with the movable page isolation steps. */ -if (unlikely (!trylock_page (page))) - goto out_putpage; +if (unlikely (!folio_trylock (folio))) + goto out_put;
I don't know much about callers that this is trying to avoid race aginst...
But for this to make sense, I think *every users* that doing their stuff with sub-page of a compound page should acquire folio lock and not page lock of sub-page, right?
Quoted 21 lines
mm/migrate.c +4
-4if (!PageMovable (page) ||PageIsolated (page)) goto out_no_isolated;@@ -106,14 +114,14 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)/* Driver shouldn't use PG_isolated bit of page->flags */ WARN_ON_ONCE (PageIsolated (page));SetPageIsolated (page); -unlock_page (page); +folio_unlock (folio);return 0 ; out_no_isolated: -unlock_page (page); -out_putpage: -put_page (page); +folio_unlock (folio); +out_put: +folio_put (folio); out:return -EBUSY ; }
Signature · 2 lines
Thanks, Hyeonggon
Quoted 7 lines · Hyeonggon Yoo
Quoted 2 lines
It's the acquisition of
the refcount which stabilises the slab flag, not holding the lock.But can you please elaborate how this prevents race between allocation & initialization of a slab and
isolate_movable_page()?Or maybe we can handle it with frozen folio as Vlastimil suggested? ;-)
Yes, we discussed that a little yesterday. I'm hoping to have a refreshed patchset for frozen folios out today. Some of this patch is still needed, even if we go that route.
Quoted 11 lines
mm/migrate.c int isolate_movable_page(struct page *page, isolate_mode_t mode) · line 103
Quoted 8 lines
mm/migrate.c +2
-2@@ -91,8 +99,8 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)* lets be sure we have the page lock * before proceeding with the movable page isolation steps. */ -if (unlikely (!trylock_page (page))) - goto out_putpage; +if (unlikely (!folio_trylock (folio))) + goto out_put;I don't know much about callers that this is trying to avoid race aginst...
Quoted 3 lines
But for this to make sense, I think *every users* that doing their stuff with sub-page of a compound page should acquire folio lock and not page lock of sub-page, right?
There is no page lock per se. If you try to acquire the lock on a tail page, it acquires the lock on its head page. It's been that way for a very long time. A lot of people are confused by this, which was part of the motivation for making it explicit with folios.
Quoted 4 lines · Matthew Wilcox
Quoted 3 lines · Hyeonggon Yoo
Quoted 2 lines
It's the acquisition of
the refcount which stabilises the slab flag, not holding the lock.
Quoted 8 lines
Quoted 4 lines
But can you please elaborate how this prevents race between allocation & initialization of a slab and
isolate_movable_page()?Or maybe we can handle it with frozen folio as Vlastimil suggested? ;-)
Yes, we discussed that a little yesterday. I'm hoping to have a refreshed patchset for frozen folios out today. Some of this patch is still needed, even if we go that route.
Good to hear that.
With that, everyting looks sane to me.
Quoted 11 lines
Quoted 11 lines
mm/migrate.c int isolate_movable_page(struct page *page, isolate_mode_t mode) · line 103
Quoted 8 lines
mm/migrate.c +2
-2@@ -91,8 +99,8 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)* lets be sure we have the page lock * before proceeding with the movable page isolation steps. */ -if (unlikely (!trylock_page (page))) - goto out_putpage; +if (unlikely (!folio_trylock (folio))) + goto out_put;I don't know much about callers that this is trying to avoid race aginst...
Quoted 8 lines
Quoted 3 lines
But for this to make sense, I think *every users* that doing their stuff with sub-page of a compound page should acquire folio lock and not page lock of sub-page, right?
There is no page lock per se. If you try to acquire the lock on a tail page, it acquires the lock on its head page. It's been that way for a very long time. A lot of people are confused by this, which was part of the motivation for making it explicit with folios.
You are right! Reading the code, too bad
I even assumed that there was sub-page lock.
Signature · 2 lines
Thanks, Hyeonggon
Quoted 27 lines · Matthew Wilcox
Quoted 7 lines · Hyeonggon Yoo
Just one more thing,
rcu_leak_callbacktoo. RCU seem to use it internally to catch doublecall_rcu().And some suggestions:
- what about adding runtimeWARN()on slab init code to catch unexpectedarch/toolchainissues?
- instead of 4, we may use macro definition? like (PAGE_MAPPING_FLAGS+ 1)?I think the real problem here is that
isolate_movable_page()is insufficiently paranoid. Looking at the gyrations that GUP and the page cache do to convince themselves that the page they got really is the page they wanted, there are a few missing pieces (eg checking that you actually got a refcount on _this_ page and not some random other page you were temporarily part of a compound page with).This patch does three things:
- Turns one of the comments into English. There are some others which I'm still scratching my head over.
- Uses a folio to help distinguish which operations are being done to the head vs the specific page (this is somewhat an abuse of the folio concept, but it's acceptable)
- Add the aforementioned check that we're actually operating on the page that we think we want to be.
- Add a check that the folio isn't secretly a slab.
Quoted 4 lines
We could put the slab check in PageMapping and call it after taking the folio lock, but that seems pointless. It's the acquisition of the refcount which stabilises the slab flag, not holding the lock.
I would like to have a working safe version in -next, even if we are able simplify it later thanks to frozen refcounts. I've made a formal patch of yours, but I'm still convinced the slab check needs to be more paranoid so it can't observe a false positive __folio_test_movable() while missing the folio_test_slab(), hence I added the barriers as in my previous attempt [1]. Does that work for you and can I add your S-o-b?
[1] https://lore.kernel.org/all/[email protected]/
----8<----
From 1d481f279f07d332ea381dfd6247a292ad403ed6 Mon Sep 17 00:00:00 2001
From: Matthew Wilcox <[email protected]>
Date: Mon, 24 Oct 2022 16:11:27 +0200
Subject: [PATCH] mm/migrate: make isolate_movable_page() skip slab pages
In the next commit we want to rearrange struct slab fields to allow a larger rcu_head. Afterwards, the page->mapping field will overlap with SLUB's "struct list_head slab_list", where the value of prev pointer can become LIST_POISON2, which is 0x122 + POISON_POINTER_DELTA. Unfortunately the bit 1 being set can confuse PageMovable() to be a false positive and cause a GPF as reported by lkp [1].
I think the real problem here is that isolate_movable_page() is insufficiently paranoid. Looking at the gyrations that GUP and the page cache do to convince themselves that the page they got really is the page they wanted, there are a few missing pieces (eg checking that you actually got a refcount on _this_ page and not some random other page you were temporarily part of a compound page with).
This patch does three things:
- Turns one of the comments into English. There are some others which I'm still scratching my head over.
- Uses a folio to help distinguish which operations are being done to the head vs the specific page (this is somewhat an abuse of the folio concept, but it's acceptable)
- Add the aforementioned check that we're actually operating on the page that we think we want to be.
- Add a check that the folio isn't secretly a slab.
We could put the slab check in PageMapping and call it after taking the folio lock, but that seems pointless. It's the acquisition of the refcount which stabilises the slab flag, not holding the lock.
[ [email protected]: add memory barriers to SLAB and SLUB's page allocation
and freeing, and their counterparts to isolate_movable_page(), to make the checks for folio_test_slab() and __folio_test_movable() SMP safe ]
[1] https://lore.kernel.org/all/[email protected]/
--- mm/migrate.c | 38 ++++++++++++++++++++++++++------------ mm/slab.c | 6 +++++- mm/slub.c | 6 +++++- 3 files changed, 36 insertions(+), 14 deletions(-)
mm/migrate.c +26 -12
@@ -60,6 +60,7 @@int isolate_movable_page (struct page *page,isolate_mode_t mode) { +struct folio *folio =page_folio (page);const struct movable_operations *mops;/*
@@ -71,16 +72,29 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)* the put_page() at the end of this block will take care of * release this page, thus avoiding a nasty leakage. */ -if (unlikely (!get_page_unless_zero (page))) +if (unlikely (!folio_try_get (folio))) goto out; +/* Recheck the page is still part of the folio we just got */ +if (unlikely (page_folio (page) != folio)) + goto out_put; + +if (unlikely (folio_test_slab (folio))) + goto out_put; +/* Pairs with smp_wmb() in slab freeing, e.g. SLUB's __free_slab() */ +smp_rmb ();/* -* Check PageMovable before holding a PG_lock because page's owner -* assumes anybody doesn't touch PG_lock of newly allocated page -* so unconditionally grabbing the lock ruins page's owner side. +* Check movable flag before taking the folio lock because +* we use non-atomic bitops on newly allocated page flags so +* unconditionally grabbing the lock ruins page's owner side. */ -if (unlikely (!__PageMovable (page))) - goto out_putpage; +if (unlikely (!__folio_test_movable (folio))) + goto out_put; +/* Pairs with smp_wmb() in slab allocation, e.g. SLUB's alloc_slab_page() */ +smp_rmb (); +if (unlikely (folio_test_slab (folio))) + goto out_put; +/* * As movable pages are not isolated from LRU lists, concurrent * compaction threads can race against page migration functions
@@ -92,8 +106,8 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)* lets be sure we have the page lock * before proceeding with the movable page isolation steps. */ -if (unlikely (!trylock_page (page))) - goto out_putpage; +if (unlikely (!folio_trylock (folio))) + goto out_put;if (!PageMovable (page) ||PageIsolated (page)) goto out_no_isolated;
@@ -107,14 +121,14 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)/* Driver shouldn't use PG_isolated bit of page->flags */ WARN_ON_ONCE (PageIsolated (page));SetPageIsolated (page); -unlock_page (page); +folio_unlock (folio);return 0 ; out_no_isolated: -unlock_page (page); -out_putpage: -put_page (page); +folio_unlock (folio); +out_put: +folio_put (folio); out:return -EBUSY ; }
mm/slab.c +5 -1
@@ -1370,6 +1370,8 @@ static struct slab *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,account_slab (slab, cachep->gfporder, cachep, flags);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();/* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks () &&page_is_pfmemalloc (folio_page (folio,0 )))slab_set_pfmemalloc (slab);
@@ -1387,9 +1389,11 @@ static void kmem_freepages(struct kmem_cache *cachep, struct slab *slab)BUG_ON (!folio_test_slab (folio));__slab_clear_pfmemalloc (slab); -__folio_clear_slab (folio);page_mapcount_reset (folio_page (folio,0 )); folio->mapping =NULL ; +/* Make the mapping reset visible before clearing the flag */ +smp_wmb (); +__folio_clear_slab (folio);if (current->reclaim_state) current->reclaim_state->reclaimed_slab +=1 << order;
mm/slub.c +5 -1
@@ -1800,6 +1800,8 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node, slab =folio_slab (folio);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();if (page_is_pfmemalloc (folio_page (folio,0 )))slab_set_pfmemalloc (slab);
@@ -2008,8 +2010,10 @@ static void __free_slab(struct kmem_cache *s, struct slab *slab) }__slab_clear_pfmemalloc (slab); -__folio_clear_slab (folio); folio->mapping =NULL ; +/* Make the mapping reset visible before clearing the flag */ +smp_wmb (); +__folio_clear_slab (folio);if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages;unaccount_slab (slab, order, s);
Signature · 1 line
2.38.0
Quoted 6 lines · Vlastimil Babka
I would like to have a working safe version in -next, even if we are able simplify it later thanks to frozen refcounts. I've made a formal patch of yours, but I'm still convinced the slab check needs to be more paranoid so it can't observe a false positive
__folio_test_movable()while missing thefolio_test_slab(), hence I added the barriers as in my previous attempt [1]. Does that work for you and can I add your S-o-b?
Thanks for picking this back up.
mm/slab.c static struct slab *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, · line 1374
Quoted 7 lines
+++
b/mm/slab.cmm/slab.c +2
-0@@ -1370,6 +1370,8 @@ static struct slab *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,account_slab (slab, cachep->gfporder, cachep, flags);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();
So what's the point of using __folio_set_slab() only to call smp_wmb() afterwards? If we call folio_set_slab() instead, don't all the other barriers go away? (This is a genuine question; I am bad at this kind of reasoning). Obviously it would still need a comment.
Quoted 13 lines · Matthew Wilcox
Quoted 6 lines · Vlastimil Babka
I would like to have a working safe version in -next, even if we are able simplify it later thanks to frozen refcounts. I've made a formal patch of yours, but I'm still convinced the slab check needs to be more paranoid so it can't observe a false positive
__folio_test_movable()while missing thefolio_test_slab(), hence I added the barriers as in my previous attempt [1]. Does that work for you and can I add your S-o-b?Thanks for picking this back up.
Quoted 3 lines
+++
b/mm/slab.c(quoted diff) +0
-0@@ -1370,6 +1370,8 @@ static struct slab *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,
Quoted 9 lines
mm/slab.c static struct slab *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, · line 1374
Quoted 4 lines
mm/slab.c +2
-0account_slab (slab, cachep->gfporder, cachep, flags);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();So what's the point of using
__folio_set_slab()only to callsmp_wmb()afterwards? If we callfolio_set_slab()instead, don't all the other barriers go away? (This is a genuine question; I am bad at this kind of reasoning). Obviously it would still need a comment.
AFAIU (which doesn't mean much, TBH :)) folio_set_slab() makes the setting of the flag protected against other flags set operations so our setting is not lost in a non-atomic RMW. But as we are the only one who can be setting any page/folio flag here (isolate_movable_page() for sure doesn't), we don't need it for that kind of atomicity for page/folio flags field.
And, simply changing it to folio_set_slab() would not add the sufficient smp_wmb() semantics to order the flags write visibility against a later write to the struct slab field that overlaps page->mapping. Only some atomic operations have that implicit barrier, (per Documentation/memory-barriers.txt and Documentation/atomic_bitops.txt) and set_bit() is not one of those. So we'd still need a smp_mb__after_atomic() AFAIU and at that point, doing the above seems less obscure to me.
(Of course if we had the reason to use folio_set_slab() for its own atomic guarantee, then smp_mb__after_atomic() instead of smp_wmb() would be better as on some architectures it would make the barrier no-op).
Quoted 33 lines · Vlastimil Babka
Quoted 27 lines · Matthew Wilcox
Quoted 7 lines · Hyeonggon Yoo
Just one more thing,
rcu_leak_callbacktoo. RCU seem to use it internally to catch doublecall_rcu().And some suggestions:
- what about adding runtimeWARN()on slab init code to catch unexpectedarch/toolchainissues?
- instead of 4, we may use macro definition? like (PAGE_MAPPING_FLAGS+ 1)?I think the real problem here is that
isolate_movable_page()is insufficiently paranoid. Looking at the gyrations that GUP and the page cache do to convince themselves that the page they got really is the page they wanted, there are a few missing pieces (eg checking that you actually got a refcount on _this_ page and not some random other page you were temporarily part of a compound page with).This patch does three things:
- Turns one of the comments into English. There are some others which I'm still scratching my head over.
- Uses a folio to help distinguish which operations are being done to the head vs the specific page (this is somewhat an abuse of the folio concept, but it's acceptable)
- Add the aforementioned check that we're actually operating on the page that we think we want to be.
- Add a check that the folio isn't secretly a slab.Quoted 4 lines
We could put the slab check in PageMapping and call it after taking the folio lock, but that seems pointless. It's the acquisition of the refcount which stabilises the slab flag, not holding the lock.
Quoted 6 lines
I would like to have a working safe version in -next, even if we are able simplify it later thanks to frozen refcounts. I've made a formal patch of yours, but I'm still convinced the slab check needs to be more paranoid so it can't observe a false positive
__folio_test_movable()while missing thefolio_test_slab(), hence I added the barriers as in my previous attempt [1]. Does that work for you and can I add your S-o-b?
Tentatively the series is here for anyone interested, will send it for proper review after the S-o-b is clarified.
https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git/log/?h=slab/for-6.2/fit_rcu_head
Quoted 1 line
Quoted 33 lines · Vlastimil Babka
Quoted 27 lines · Matthew Wilcox
Quoted 7 lines · Hyeonggon Yoo
Just one more thing,
rcu_leak_callbacktoo. RCU seem to use it internally to catch doublecall_rcu().And some suggestions:
- what about adding runtimeWARN()on slab init code to catch unexpectedarch/toolchainissues?
- instead of 4, we may use macro definition? like (PAGE_MAPPING_FLAGS+ 1)?I think the real problem here is that
isolate_movable_page()is insufficiently paranoid. Looking at the gyrations that GUP and the page cache do to convince themselves that the page they got really is the page they wanted, there are a few missing pieces (eg checking that you actually got a refcount on _this_ page and not some random other page you were temporarily part of a compound page with).This patch does three things:
- Turns one of the comments into English. There are some others which I'm still scratching my head over.
- Uses a folio to help distinguish which operations are being done to the head vs the specific page (this is somewhat an abuse of the folio concept, but it's acceptable)
- Add the aforementioned check that we're actually operating on the page that we think we want to be.
- Add a check that the folio isn't secretly a slab.Quoted 4 lines
We could put the slab check in PageMapping and call it after taking the folio lock, but that seems pointless. It's the acquisition of the refcount which stabilises the slab flag, not holding the lock.
Quoted 8 lines
I would like to have a working safe version in -next, even if we are able simplify it later thanks to frozen refcounts. I've made a formal patch of yours, but I'm still convinced the slab check needs to be more paranoid so it can't observe a false positive
__folio_test_movable()while missing thefolio_test_slab(), hence I added the barriers as in my previous attempt [1]. Does that work for you and can I add your S-o-b?
Ignore me, don't let me distract if you're happy with Matthew's patch (I know little of PageMovable, and I haven't tried to understand it); but it did look to me more like 6.2 material, and I was surprised that you dropped the simple align(4) approach for 6.1.
Because of Hyeonggon's rcu_leak_callback() observation? That was a good catch, but turned out to be irrelevant, because it was only for an RCU debugging option, which would never be set up on a struct page (well, maybe it would in a dynamically-allocated-struct-page future).
Hugh
Quoted 8 lines · Hugh Dickins
Quoted 2 lines · Vlastimil Babka
Ignore me, don't let me distract if you're happy with Matthew's patch (I know little of PageMovable, and I haven't tried to understand it); but it did look to me more like 6.2 material,
Yes, it is targetted towards 6.2 now.
Quoted 7 lines
and I was surprised that
you dropped the simple align(4) approach for 6.1.Because of Hyeonggon's
rcu_leak_callback()observation? That was a good catch, but turned out to be irrelevant, because it was only for an RCU debugging option, which would never be set up on a struct page (well, maybe it would in a dynamically-allocated-struct-page future).
It was mainly due to David's observation:
https://lore.kernel.org/all/[email protected]/
I did also conclude that align(4) indeed overrides anything set via CONFIG_FUNCTION_ALIGNMENT (and passed by -falign-functions) in a way that a globally configured larger alignment can be made smaller by this macro, and it was all too risky and last minute to me, while Joel's series wasn't targetted to 6.1 anyway so there was no urgency.
And yeah it looks like a gcc bug to me.
Quoted 1 line
Hugh
On Mon, Oct 24, 2022 at 04:35:04PM +0200, Vlastimil Babka wrote:
[,,,]
Quoted 188 lines
I would like to have a working safe version in -next, even if we are able simplify it later thanks to frozen refcounts. I've made a formal patch of yours, but I'm still convinced the slab check needs to be more paranoid so it can't observe a false positive
__folio_test_movable()while missing thefolio_test_slab(), hence I added the barriers as in my previous attempt [1]. Does that work for you and can I add your S-o-b?[1] https://lore.kernel.org/all/[email protected]/
----8<----
From1d481f279f07d332ea381dfd6247a292ad403ed6Mon Sep 17 00:00:00 2001
From: Matthew Wilcox <[email protected]>
Date: Mon, 24 Oct 2022 16:11:27 +0200
Subject: [PATCH] mm/migrate: makeisolate_movable_page()skip slab pagesIn the next commit we want to rearrange struct slab fields to allow a larger
rcu_head. Afterwards, thepage->mappingfield will overlap with SLUB's "structlist_headslab_list", where the value of prev pointer can becomeLIST_POISON2, which is0x122+POISON_POINTER_DELTA. Unfortunately the bit 1 being set can confusePageMovable()to be a false positive and cause a GPF as reported by lkp [1].I think the real problem here is that
isolate_movable_page()is insufficiently paranoid. Looking at the gyrations that GUP and the page cache do to convince themselves that the page they got really is the page they wanted, there are a few missing pieces (eg checking that you actually got a refcount on _this_ page and not some random other page you were temporarily part of a compound page with).This patch does three things:
- Turns one of the comments into English. There are some others which I'm still scratching my head over.
- Uses a folio to help distinguish which operations are being done to the head vs the specific page (this is somewhat an abuse of the folio concept, but it's acceptable)
- Add the aforementioned check that we're actually operating on the page that we think we want to be.
- Add a check that the folio isn't secretly a slab.We could put the slab check in PageMapping and call it after taking the folio lock, but that seems pointless. It's the acquisition of the refcount which stabilises the slab flag, not holding the lock.
[ [email protected]: add memory barriers to SLAB and SLUB's page allocation
and freeing, and their counterparts toisolate_movable_page(), to make the checks forfolio_test_slab()and__folio_test_movable()SMP safe ][1] https://lore.kernel.org/all/[email protected]/
Signed-off-by Vlastimil Babka <[email protected]>--- mm/migrate.c | 38 ++++++++++++++++++++++++++------------ mm/slab.c | 6 +++++- mm/slub.c | 6 +++++- 3 files changed, 36 insertions(+), 14 deletions(-)mm/migrate.c +26
-12@@ -60,6 +60,7 @@int isolate_movable_page (struct page *page,isolate_mode_t mode) { +struct folio *folio =page_folio (page);const struct movable_operations *mops;/* @@ -71,16 +72,29 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)* the put_page() at the end of this block will take care of * release this page, thus avoiding a nasty leakage. */ -if (unlikely (!get_page_unless_zero (page))) +if (unlikely (!folio_try_get (folio))) goto out; +/* Recheck the page is still part of the folio we just got */ +if (unlikely (page_folio (page) != folio)) + goto out_put; + +if (unlikely (folio_test_slab (folio))) + goto out_put; +/* Pairs with smp_wmb() in slab freeing, e.g. SLUB's __free_slab() */ +smp_rmb ();/* -* Check PageMovable before holding a PG_lock because page's owner -* assumes anybody doesn't touch PG_lock of newly allocated page -* so unconditionally grabbing the lock ruins page's owner side. +* Check movable flag before taking the folio lock because +* we use non-atomic bitops on newly allocated page flags so +* unconditionally grabbing the lock ruins page's owner side. */ -if (unlikely (!__PageMovable (page))) - goto out_putpage; +if (unlikely (!__folio_test_movable (folio))) + goto out_put; +/* Pairs with smp_wmb() in slab allocation, e.g. SLUB's alloc_slab_page() */ +smp_rmb (); +if (unlikely (folio_test_slab (folio))) + goto out_put; +/* * As movable pages are not isolated from LRU lists, concurrent * compaction threads can race against page migration functions @@ -92,8 +106,8 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)* lets be sure we have the page lock * before proceeding with the movable page isolation steps. */ -if (unlikely (!trylock_page (page))) - goto out_putpage; +if (unlikely (!folio_trylock (folio))) + goto out_put;if (!PageMovable (page) ||PageIsolated (page)) goto out_no_isolated;@@ -107,14 +121,14 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)/* Driver shouldn't use PG_isolated bit of page->flags */ WARN_ON_ONCE (PageIsolated (page));SetPageIsolated (page); -unlock_page (page); +folio_unlock (folio);return 0 ; out_no_isolated: -unlock_page (page); -out_putpage: -put_page (page); +folio_unlock (folio); +out_put: +folio_put (folio); out:return -EBUSY ; }mm/slab.c +5
-1@@ -1370,6 +1370,8 @@ static struct slab *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,account_slab (slab, cachep->gfporder, cachep, flags);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();/* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks () &&page_is_pfmemalloc (folio_page (folio,0 )))slab_set_pfmemalloc (slab);@@ -1387,9 +1389,11 @@ static void kmem_freepages(struct kmem_cache *cachep, struct slab *slab)BUG_ON (!folio_test_slab (folio));__slab_clear_pfmemalloc (slab); -__folio_clear_slab (folio);page_mapcount_reset (folio_page (folio,0 )); folio->mapping =NULL ; +/* Make the mapping reset visible before clearing the flag */ +smp_wmb (); +__folio_clear_slab (folio);if (current->reclaim_state) current->reclaim_state->reclaimed_slab +=1 << order;mm/slub.c +5
-1@@ -1800,6 +1800,8 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node, slab =folio_slab (folio);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();if (page_is_pfmemalloc (folio_page (folio,0 )))slab_set_pfmemalloc (slab);@@ -2008,8 +2010,10 @@ static void __free_slab(struct kmem_cache *s, struct slab *slab) }__slab_clear_pfmemalloc (slab); -__folio_clear_slab (folio); folio->mapping =NULL ; +/* Make the mapping reset visible before clearing the flag */ +smp_wmb (); +__folio_clear_slab (folio);if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages;unaccount_slab (slab, order, s);Signature · 1 line
2.38.0
Do we need to try this with memory barriers before frozen refcount lands in? It's quite complicated and IIUC there is a still theoretical race:
At isolate_movable_page: At slab alloc: At slab free:
folio = alloc_pages(flags, order)
folio_try_get()folio_test_slab() == false
__folio_set_slab(folio)
smp_wmb()
call_rcu(&slab->rcu_head, rcu_free_slab);smp_rmb()__folio_test_movable() == true
folio->mapping = NULL;
smp_wmb()
__folio_clear_slab(folio);smp_rmb()folio_test_slab() == false
folio_trylock()mops->isolate_page() (*crash*)
Please let me know if I'm missing something ;-)
Thanks!
Signature · 1 line
Hyeonggon
Quoted 46 lines · Hyeonggon Yoo
On Mon, Oct 24, 2022 at 04:35:04PM +0200, Vlastimil Babka wrote:
[,,,]
Quoted 42 lines
mm/slab.c +5
-1@@ -1370,6 +1370,8 @@ static struct slab *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,account_slab (slab, cachep->gfporder, cachep, flags);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();/* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks () &&page_is_pfmemalloc (folio_page (folio,0 )))slab_set_pfmemalloc (slab);@@ -1387,9 +1389,11 @@ static void kmem_freepages(struct kmem_cache *cachep, struct slab *slab)BUG_ON (!folio_test_slab (folio));__slab_clear_pfmemalloc (slab); -__folio_clear_slab (folio);page_mapcount_reset (folio_page (folio,0 )); folio->mapping =NULL ; +/* Make the mapping reset visible before clearing the flag */ +smp_wmb (); +__folio_clear_slab (folio);if (current->reclaim_state) current->reclaim_state->reclaimed_slab +=1 << order;mm/slub.c +2
-0@@ -1800,6 +1800,8 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node, slab =folio_slab (folio);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();if (page_is_pfmemalloc (folio_page (folio,0 )))slab_set_pfmemalloc (slab);@@ -2008,8 +2010,10 @@ static void __free_slab(struct kmem_cache *s, struct slab *slab) }
Quoted 13 lines
Quoted 11 lines
mm/slub.c +3
-1__slab_clear_pfmemalloc (slab); -__folio_clear_slab (folio); folio->mapping =NULL ; +/* Make the mapping reset visible before clearing the flag */ +smp_wmb (); +__folio_clear_slab (folio);if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages;unaccount_slab (slab, order, s);Signature · 1 line
2.38.0Do we need to try this with memory barriers before frozen refcount lands in?
There was IIRC an unresolved issue with frozen refcount tripping the page isolation code so I didn't want to be depending on that.
Quoted 16 lines
It's quite complicated and IIUC there is a still theoretical race:
At
isolate_movable_page: At slab alloc: At slab free:
folio =alloc_pages(flags, order)
folio_try_get()folio_test_slab()== false__folio_set_slab(folio) smp_wmb() call_rcu(&slab->rcu_head, rcu_free_slab);
smp_rmb()__folio_test_movable()== true
Quoted 7 lines
folio->mapping = NULL; smp_wmb() __folio_clear_slab(folio);
smp_rmb()folio_test_slab()== false
folio_trylock()
There's also between above and below:
if (!PageMovable(page) || PageIsolated(page))
goto out_no_isolated;
mops = page_movable_ops(page);
If we put another smp_rmb() before the PageMovable test, could that have helped? It would assure we observe the folio->mapping = NULL; from the "slab free" side?
But yeah, it's getting ridiculous. Maybe there's a simpler way to check two bits in two different bytes atomically. Or maybe it's just an impossible task, I feel I just dunno computers at this point.
Quoted 6 lines
mops->isolate_page()(*crash*)Please let me know if I'm missing something ;-)
Thanks!
Quoted 11 lines · Vlastimil Babka
Quoted 8 lines · Hugh Dickins
Quoted 2 lines · Vlastimil Babka
Ignore me, don't let me distract if you're happy with Matthew's patch (I know little of PageMovable, and I haven't tried to understand it); but it did look to me more like 6.2 material,
Yes, it is targetted towards 6.2 now.
That's good.
Quoted 11 lines
Quoted 8 lines
and I was surprised that
you dropped the simple align(4) approach for 6.1.Because of Hyeonggon's
rcu_leak_callback()observation? That was a good catch, but turned out to be irrelevant, because it was only for an RCU debugging option, which would never be set up on a struct page (well, maybe it would in a dynamically-allocated-struct-page future).It was mainly due to David's observation:
Quoted 7 lines
https://lore.kernel.org/all/[email protected]/
I did also conclude that align(4) indeed overrides anything set via
CONFIG_FUNCTION_ALIGNMENT(and passed by -falign-functions) in a way that a globally configured larger alignment can be made smaller by this macro, and it was all too risky and last minute to me, while Joel's series wasn't targetted to 6.1 anyway so there was no urgency.
Oh, I had thought it was heading for 6.1. Yes, you have good reason to drop the alignment trick there - thanks for verifying David's point. I hadn't seen followup, and assumed that the earlier i915 example gave assurance that alignment was usable (but maybe that's part of why the i915 folks stopped doing it that way).
Thanks,
Hugh
Quoted 4 lines
And yeah it looks like a gcc bug to me.
Quoted 1 line
Hugh
Quoted 95 lines · Vlastimil Babka
Quoted 46 lines · Hyeonggon Yoo
On Mon, Oct 24, 2022 at 04:35:04PM +0200, Vlastimil Babka wrote:
[,,,]
Quoted 42 lines
mm/slab.c +5
-1@@ -1370,6 +1370,8 @@ static struct slab *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,account_slab (slab, cachep->gfporder, cachep, flags);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();/* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks () &&page_is_pfmemalloc (folio_page (folio,0 )))slab_set_pfmemalloc (slab);@@ -1387,9 +1389,11 @@ static void kmem_freepages(struct kmem_cache *cachep, struct slab *slab)BUG_ON (!folio_test_slab (folio));__slab_clear_pfmemalloc (slab); -__folio_clear_slab (folio);page_mapcount_reset (folio_page (folio,0 )); folio->mapping =NULL ; +/* Make the mapping reset visible before clearing the flag */ +smp_wmb (); +__folio_clear_slab (folio);if (current->reclaim_state) current->reclaim_state->reclaimed_slab +=1 << order;mm/slub.c +2
-0@@ -1800,6 +1800,8 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node, slab =folio_slab (folio);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();if (page_is_pfmemalloc (folio_page (folio,0 )))slab_set_pfmemalloc (slab);@@ -2008,8 +2010,10 @@ static void __free_slab(struct kmem_cache *s, struct slab *slab) }Quoted 13 lines
Quoted 11 lines
mm/slub.c +3
-1__slab_clear_pfmemalloc (slab); -__folio_clear_slab (folio); folio->mapping =NULL ; +/* Make the mapping reset visible before clearing the flag */ +smp_wmb (); +__folio_clear_slab (folio);if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages;unaccount_slab (slab, order, s);Signature · 1 line
2.38.0Do we need to try this with memory barriers before frozen refcount lands in?
There was IIRC an unresolved issue with frozen refcount tripping the page isolation code so I didn't want to be depending on that.
Quoted 16 lines
It's quite complicated and IIUC there is a still theoretical race:
At
isolate_movable_page: At slab alloc: At slab free:
folio =alloc_pages(flags, order)
folio_try_get()folio_test_slab()== false__folio_set_slab(folio) smp_wmb() call_rcu(&slab->rcu_head, rcu_free_slab);
smp_rmb()__folio_test_movable()== trueQuoted 7 lines
folio->mapping = NULL; smp_wmb() __folio_clear_slab(folio);
smp_rmb()folio_test_slab()== false
folio_trylock()There's also between above and below:
if (!PageMovable(page) || PageIsolated(page))
goto out_no_isolated;mops =
page_movable_ops(page);
Quoted 7 lines
If we put another
smp_rmb()before the PageMovable test, could that have helped? It would assure we observe thefolio->mapping= NULL; from the "slab free" side?But yeah, it's getting ridiculous. Maybe there's a simpler way to check two bits in two different bytes atomically. Or maybe it's just an impossible task, I feel I just dunno computers at this point.
After more thought, I think I just made a mistake by doing two folio_test_slab() tests around a single __folio_test_movable(). What I was supposed to do was two __folio_test_movable() tests around a single folio_test_slab()... I hope. That should take care of your scenario, or do you see another one? Thanks.
----8----
From 5ca1c10f6411d73ad579b58d4fa10326bf77cf0a Mon Sep 17 00:00:00 2001
From: Matthew Wilcox <[email protected]>
Date: Mon, 24 Oct 2022 16:11:27 +0200
Subject: [PATCH] mm/migrate: make isolate_movable_page() skip slab pages
In the next commit we want to rearrange struct slab fields to allow a larger rcu_head. Afterwards, the page->mapping field will overlap with SLUB's "struct list_head slab_list", where the value of prev pointer can become LIST_POISON2, which is 0x122 + POISON_POINTER_DELTA. Unfortunately the bit 1 being set can confuse PageMovable() to be a false positive and cause a GPF as reported by lkp [1].
I think the real problem here is that isolate_movable_page() is insufficiently paranoid. Looking at the gyrations that GUP and the page cache do to convince themselves that the page they got really is the page they wanted, there are a few missing pieces (eg checking that you actually got a refcount on _this_ page and not some random other page you were temporarily part of a compound page with).
This patch does three things:
- Turns one of the comments into English. There are some others which I'm still scratching my head over.
- Uses a folio to help distinguish which operations are being done to the head vs the specific page (this is somewhat an abuse of the folio concept, but it's acceptable)
- Add the aforementioned check that we're actually operating on the page that we think we want to be.
- Add a check that the folio isn't secretly a slab.
We could put the slab check in PageMapping and call it after taking the folio lock, but that seems pointless. It's the acquisition of the refcount which stabilises the slab flag, not holding the lock.
[ [email protected]: add memory barriers to SLAB and SLUB's page allocation
and freeing, and their counterparts to isolate_movable_page(), to make the checks for folio_test_slab() and __folio_test_movable() SMP safe ]
[1] https://lore.kernel.org/all/[email protected]/
--- mm/migrate.c | 40 ++++++++++++++++++++++++++++------------ mm/slab.c | 6 +++++- mm/slub.c | 6 +++++- 3 files changed, 38 insertions(+), 14 deletions(-)
mm/migrate.c +28 -12
@@ -60,6 +60,7 @@int isolate_movable_page (struct page *page,isolate_mode_t mode) { +struct folio *folio =page_folio (page);const struct movable_operations *mops;/*
@@ -71,16 +72,31 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)* the put_page() at the end of this block will take care of * release this page, thus avoiding a nasty leakage. */ -if (unlikely (!get_page_unless_zero (page))) +if (unlikely (!folio_try_get (folio))) goto out; +/* Recheck the page is still part of the folio we just got */ +if (unlikely (page_folio (page) != folio)) + goto out_put; +/* -* Check PageMovable before holding a PG_lock because page's owner -* assumes anybody doesn't touch PG_lock of newly allocated page -* so unconditionally grabbing the lock ruins page's owner side. +* Check movable flag before taking the folio lock because +* we use non-atomic bitops on newly allocated page flags so +* unconditionally grabbing the lock ruins page's owner side. +* Make sure we don't have a slab folio here as its usage of the +* mapping field can cause a false positive movable flag. */ -if (unlikely (!__PageMovable (page))) - goto out_putpage; +if (unlikely (!__folio_test_movable (folio))) + goto out_put; +/* Pairs with smp_wmb() in slab allocation, e.g. SLUB's alloc_slab_page() */ +smp_rmb (); +if (unlikely (folio_test_slab (folio))) + goto out_put; +/* Pairs with smp_wmb() in slab freeing, e.g. SLUB's __free_slab() */ +smp_rmb (); +if (unlikely (!__folio_test_movable (folio))) + goto out_put; +/* * As movable pages are not isolated from LRU lists, concurrent * compaction threads can race against page migration functions
@@ -92,8 +108,8 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)* lets be sure we have the page lock * before proceeding with the movable page isolation steps. */ -if (unlikely (!trylock_page (page))) - goto out_putpage; +if (unlikely (!folio_trylock (folio))) + goto out_put;if (!PageMovable (page) ||PageIsolated (page)) goto out_no_isolated;
@@ -107,14 +123,14 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)/* Driver shouldn't use PG_isolated bit of page->flags */ WARN_ON_ONCE (PageIsolated (page));SetPageIsolated (page); -unlock_page (page); +folio_unlock (folio);return 0 ; out_no_isolated: -unlock_page (page); -out_putpage: -put_page (page); +folio_unlock (folio); +out_put: +folio_put (folio); out:return -EBUSY ; }
mm/slab.c +5 -1
@@ -1370,6 +1370,8 @@ static struct slab *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,account_slab (slab, cachep->gfporder, cachep, flags);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();/* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks () &&page_is_pfmemalloc (folio_page (folio,0 )))slab_set_pfmemalloc (slab);
@@ -1387,9 +1389,11 @@ static void kmem_freepages(struct kmem_cache *cachep, struct slab *slab)BUG_ON (!folio_test_slab (folio));__slab_clear_pfmemalloc (slab); -__folio_clear_slab (folio);page_mapcount_reset (folio_page (folio,0 )); folio->mapping =NULL ; +/* Make the mapping reset visible before clearing the flag */ +smp_wmb (); +__folio_clear_slab (folio);if (current->reclaim_state) current->reclaim_state->reclaimed_slab +=1 << order;
mm/slub.c +5 -1
@@ -1800,6 +1800,8 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node, slab =folio_slab (folio);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();if (page_is_pfmemalloc (folio_page (folio,0 )))slab_set_pfmemalloc (slab);
@@ -2000,8 +2002,10 @@ static void __free_slab(struct kmem_cache *s, struct slab *slab)int pages =1 << order;__slab_clear_pfmemalloc (slab); -__folio_clear_slab (folio); folio->mapping =NULL ; +/* Make the mapping reset visible before clearing the flag */ +smp_wmb (); +__folio_clear_slab (folio);if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages;unaccount_slab (slab, order, s);
Signature · 1 line
2.38.0
Quoted 62 lines · Vlastimil Babka
Quoted 61 lines · Vlastimil Babka
Quoted 59 lines · Hyeonggon Yoo
On Mon, Oct 24, 2022 at 04:35:04PM +0200, Vlastimil Babka wrote:
[,,,]
Quoted 53 lines
mm/slab.c +5
-1@@ -1370,6 +1370,8 @@ static struct slab *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,account_slab (slab, cachep->gfporder, cachep, flags);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();/* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks () &&page_is_pfmemalloc (folio_page (folio,0 )))slab_set_pfmemalloc (slab);@@ -1387,9 +1389,11 @@ static void kmem_freepages(struct kmem_cache *cachep, struct slab *slab)BUG_ON (!folio_test_slab (folio));__slab_clear_pfmemalloc (slab); -__folio_clear_slab (folio);page_mapcount_reset (folio_page (folio,0 )); folio->mapping =NULL ; +/* Make the mapping reset visible before clearing the flag */ +smp_wmb (); +__folio_clear_slab (folio);if (current->reclaim_state) current->reclaim_state->reclaimed_slab +=1 << order;mm/slub.c +5
-1@@ -1800,6 +1800,8 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node, slab =folio_slab (folio);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();if (page_is_pfmemalloc (folio_page (folio,0 )))slab_set_pfmemalloc (slab);@@ -2008,8 +2010,10 @@ static void __free_slab(struct kmem_cache *s, struct slab *slab) }__slab_clear_pfmemalloc (slab); -__folio_clear_slab (folio); folio->mapping =NULL ; +/* Make the mapping reset visible before clearing the flag */ +smp_wmb (); +__folio_clear_slab (folio);if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages;unaccount_slab (slab, order, s);Signature · 1 line
2.38.0Do we need to try this with memory barriers before frozen refcount lands in?
Quoted 2 lines
Quoted 2 lines
There was IIRC an unresolved issue with frozen refcount tripping the page isolation code so I didn't want to be depending on that.
Understood.
Quoted 39 lines
Quoted 31 lines
Quoted 16 lines
It's quite complicated and IIUC there is a still theoretical race:
At
isolate_movable_page: At slab alloc: At slab free:
folio =alloc_pages(flags, order)
folio_try_get()folio_test_slab()== false__folio_set_slab(folio) smp_wmb() call_rcu(&slab->rcu_head, rcu_free_slab);
smp_rmb()__folio_test_movable()== trueQuoted 7 lines
folio->mapping = NULL; smp_wmb() __folio_clear_slab(folio);
smp_rmb()folio_test_slab()== false
folio_trylock()There's also between above and below:
if (!PageMovable(page) || PageIsolated(page))
goto out_no_isolated;mops =
page_movable_ops(page);Quoted 7 lines
If we put another
smp_rmb()before the PageMovable test, could that have helped? It would assure we observe thefolio->mapping= NULL; from the "slab free" side?But yeah, it's getting ridiculous. Maybe there's a simpler way to check two bits in two different bytes atomically. Or maybe it's just an impossible task, I feel I just dunno computers at this point.
Quoted 5 lines
After more thought, I think I just made a mistake by doing two
folio_test_slab()tests around a single__folio_test_movable(). What I was supposed to do was two__folio_test_movable()tests around a singlefolio_test_slab()... I hope. That should take care of your scenario, or do you see another one? Thanks.
I couldn't find one with this patch.
To best my understanding this looks at least correct to me.
Hope there is no other place that is confused
by anonymous checking for random page like here.
Thanks.
Quoted 181 lines
----8----
From5ca1c10f6411d73ad579b58d4fa10326bf77cf0aMon Sep 17 00:00:00 2001
From: Matthew Wilcox <[email protected]>
Date: Mon, 24 Oct 2022 16:11:27 +0200
Subject: [PATCH] mm/migrate: makeisolate_movable_page()skip slab pagesIn the next commit we want to rearrange struct slab fields to allow a larger
rcu_head. Afterwards, thepage->mappingfield will overlap with SLUB's "structlist_headslab_list", where the value of prev pointer can becomeLIST_POISON2, which is0x122+POISON_POINTER_DELTA. Unfortunately the bit 1 being set can confusePageMovable()to be a false positive and cause a GPF as reported by lkp [1].I think the real problem here is that
isolate_movable_page()is insufficiently paranoid. Looking at the gyrations that GUP and the page cache do to convince themselves that the page they got really is the page they wanted, there are a few missing pieces (eg checking that you actually got a refcount on _this_ page and not some random other page you were temporarily part of a compound page with).This patch does three things:
- Turns one of the comments into English. There are some others which I'm still scratching my head over.
- Uses a folio to help distinguish which operations are being done to the head vs the specific page (this is somewhat an abuse of the folio concept, but it's acceptable)
- Add the aforementioned check that we're actually operating on the page that we think we want to be.
- Add a check that the folio isn't secretly a slab.We could put the slab check in PageMapping and call it after taking the folio lock, but that seems pointless. It's the acquisition of the refcount which stabilises the slab flag, not holding the lock.
[ [email protected]: add memory barriers to SLAB and SLUB's page allocation
and freeing, and their counterparts toisolate_movable_page(), to make the checks forfolio_test_slab()and__folio_test_movable()SMP safe ][1] https://lore.kernel.org/all/[email protected]/
Signed-off-by Vlastimil Babka <[email protected]>--- mm/migrate.c | 40 ++++++++++++++++++++++++++++------------ mm/slab.c | 6 +++++- mm/slub.c | 6 +++++- 3 files changed, 38 insertions(+), 14 deletions(-)mm/migrate.c +28
-12@@ -60,6 +60,7 @@int isolate_movable_page (struct page *page,isolate_mode_t mode) { +struct folio *folio =page_folio (page);const struct movable_operations *mops;/* @@ -71,16 +72,31 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)* the put_page() at the end of this block will take care of * release this page, thus avoiding a nasty leakage. */ -if (unlikely (!get_page_unless_zero (page))) +if (unlikely (!folio_try_get (folio))) goto out; +/* Recheck the page is still part of the folio we just got */ +if (unlikely (page_folio (page) != folio)) + goto out_put; +/* -* Check PageMovable before holding a PG_lock because page's owner -* assumes anybody doesn't touch PG_lock of newly allocated page -* so unconditionally grabbing the lock ruins page's owner side. +* Check movable flag before taking the folio lock because +* we use non-atomic bitops on newly allocated page flags so +* unconditionally grabbing the lock ruins page's owner side. +* Make sure we don't have a slab folio here as its usage of the +* mapping field can cause a false positive movable flag. */ -if (unlikely (!__PageMovable (page))) - goto out_putpage; +if (unlikely (!__folio_test_movable (folio))) + goto out_put; +/* Pairs with smp_wmb() in slab allocation, e.g. SLUB's alloc_slab_page() */ +smp_rmb (); +if (unlikely (folio_test_slab (folio))) + goto out_put; +/* Pairs with smp_wmb() in slab freeing, e.g. SLUB's __free_slab() */ +smp_rmb (); +if (unlikely (!__folio_test_movable (folio))) + goto out_put; +/* * As movable pages are not isolated from LRU lists, concurrent * compaction threads can race against page migration functions @@ -92,8 +108,8 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)* lets be sure we have the page lock * before proceeding with the movable page isolation steps. */ -if (unlikely (!trylock_page (page))) - goto out_putpage; +if (unlikely (!folio_trylock (folio))) + goto out_put;if (!PageMovable (page) ||PageIsolated (page)) goto out_no_isolated;@@ -107,14 +123,14 @@ int isolate_movable_page(struct page *page, isolate_mode_t mode)/* Driver shouldn't use PG_isolated bit of page->flags */ WARN_ON_ONCE (PageIsolated (page));SetPageIsolated (page); -unlock_page (page); +folio_unlock (folio);return 0 ; out_no_isolated: -unlock_page (page); -out_putpage: -put_page (page); +folio_unlock (folio); +out_put: +folio_put (folio); out:return -EBUSY ; }mm/slab.c +5
-1@@ -1370,6 +1370,8 @@ static struct slab *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,account_slab (slab, cachep->gfporder, cachep, flags);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();/* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks () &&page_is_pfmemalloc (folio_page (folio,0 )))slab_set_pfmemalloc (slab);@@ -1387,9 +1389,11 @@ static void kmem_freepages(struct kmem_cache *cachep, struct slab *slab)BUG_ON (!folio_test_slab (folio));__slab_clear_pfmemalloc (slab); -__folio_clear_slab (folio);page_mapcount_reset (folio_page (folio,0 )); folio->mapping =NULL ; +/* Make the mapping reset visible before clearing the flag */ +smp_wmb (); +__folio_clear_slab (folio);if (current->reclaim_state) current->reclaim_state->reclaimed_slab +=1 << order;mm/slub.c +5
-1@@ -1800,6 +1800,8 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node, slab =folio_slab (folio);__folio_set_slab (folio); +/* Make the flag visible before any changes to folio->mapping */ +smp_wmb ();if (page_is_pfmemalloc (folio_page (folio,0 )))slab_set_pfmemalloc (slab);@@ -2000,8 +2002,10 @@ static void __free_slab(struct kmem_cache *s, struct slab *slab)int pages =1 << order;__slab_clear_pfmemalloc (slab); -__folio_clear_slab (folio); folio->mapping =NULL ; +/* Make the mapping reset visible before clearing the flag */ +smp_wmb (); +__folio_clear_slab (folio);if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages;unaccount_slab (slab, order, s);Signature · 1 line
2.38.0
Quoted 33 lines · Vlastimil Babka
Quoted 27 lines · Matthew Wilcox
Quoted 7 lines · Hyeonggon Yoo
Just one more thing,
rcu_leak_callbacktoo. RCU seem to use it internally to catch doublecall_rcu().And some suggestions:
- what about adding runtimeWARN()on slab init code to catch unexpectedarch/toolchainissues?
- instead of 4, we may use macro definition? like (PAGE_MAPPING_FLAGS+ 1)?I think the real problem here is that
isolate_movable_page()is insufficiently paranoid. Looking at the gyrations that GUP and the page cache do to convince themselves that the page they got really is the page they wanted, there are a few missing pieces (eg checking that you actually got a refcount on _this_ page and not some random other page you were temporarily part of a compound page with).This patch does three things:
- Turns one of the comments into English. There are some others which I'm still scratching my head over.
- Uses a folio to help distinguish which operations are being done to the head vs the specific page (this is somewhat an abuse of the folio concept, but it's acceptable)
- Add the aforementioned check that we're actually operating on the page that we think we want to be.
- Add a check that the folio isn't secretly a slab.Quoted 4 lines
We could put the slab check in PageMapping and call it after taking the folio lock, but that seems pointless. It's the acquisition of the refcount which stabilises the slab flag, not holding the lock.
Quoted 8 lines
I would like to have a working safe version in -next, even if we are able simplify it later thanks to frozen refcounts. I've made a formal patch of yours, but I'm still convinced the slab check needs to be more paranoid so it can't observe a false positive
__folio_test_movable()while missing thefolio_test_slab(), hence I added the barriers as in my previous attempt [1]. Does that work for you and can I add your S-o-b?
To move on, I pushed a branch based on a new version of [1] above. It lacks Matthew's folio parts, which are not IMHO that critical right now, so can be added later.
It's here: https://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab.git/log/?h=slab/for-6.2/fit_rcu_head
Will also send for formal review soon.