mm: krealloc: consider spare memory for __GFP_ZERO
Subsystems (2)
PATCH 1/2 mm: krealloc: consider spare memory for __GFP_ZERO
As long as krealloc() is called with __GFP_ZERO consistently, starting with the initial memory allocation, __GFP_ZERO should be fully honored.
However, if for an existing allocation krealloc() is called with a decreased size, it is not ensured that the spare portion the allocation is zeroed. Thus, if krealloc() is subsequently called with a larger size again, __GFP_ZERO can't be fully honored, since we don't know the previous size, but only the bucket size.
Example:
buf = kzalloc(64, GFP_KERNEL); memset(buf, 0xff, 64); buf = krealloc(buf, 48, GFP_KERNEL | __GFP_ZERO); /* After this call the last 16 bytes are still 0xff. */ buf = krealloc(buf, 64, GFP_KERNEL | __GFP_ZERO);
Fix this, by explicitly setting spare memory to zero, when shrinking an allocation with __GFP_ZERO flag set or init_on_alloc enabled.
--- mm/slab_common.c | 7 +++++++ 1 file changed, 7 insertions(+)
mm/slab_common.c +7 -0
@@ -1273,6 +1273,13 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)/* If the object still fits, repoison it precisely. */ if (ks >= new_size) { +/* Zero out spare memory. */ +if (want_init_on_alloc (flags)) { +kasan_disable_current (); +memset ((void *)p + new_size,0 , ks - new_size); +kasan_enable_current (); + } + p =kasan_krealloc ((void *)p, new_size, flags);return (void *)p; }
base-commit: 7c3dd6d99f2df6a9d7944ee8505b195ba51c9b68
Signature · 1 line
2.45.2
PATCH 2/2 mm: krealloc: clarify valid usage of __GFP_ZERO
Properly document that if __GFP_ZERO logic is requested, callers must ensure that, starting with the initial memory allocation, every subsequent call to this API for the same memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that __GFP_ZERO is not fully honored by this API.
--- include/linux/slab.h | 8 ++++++++ mm/slab_common.c | 10 ++++++++-- 2 files changed, 16 insertions(+), 2 deletions(-)
include/linux/slab.h +8 -0
@@ -733,6 +733,14 @@ static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t siz* @new_n: new number of elements to alloc * @new_size: new size of a single member of the array * @flags: the type of memory to allocate (see kmalloc) +* +* If __GFP_ZERO logic is requested, callers must ensure that, starting with the +* initial memory allocation, every subsequent call to this API for the same +* memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that +* __GFP_ZERO is not fully honored by this API. +* +* In any case, the contents of the object pointed to are preserved up to the +* lesser of the new and old sizes. */ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(void *p, size_t new_n,
mm/slab_common.c +8 -2
@@ -1301,11 +1301,17 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)* @new_size: how many bytes of memory are required. * @flags: the type of memory to allocate. * -* The contents of the object pointed to are preserved up to the -* lesser of the new and old sizes (__GFP_ZERO flag is effectively ignored). * If @p is %NULL, krealloc() behaves exactly like kmalloc(). If @new_size * is 0 and @p is not a %NULL pointer, the object pointed to is freed. * +* If __GFP_ZERO logic is requested, callers must ensure that, starting with the +* initial memory allocation, every subsequent call to this API for the same +* memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that +* __GFP_ZERO is not fully honored by this API. +* +* In any case, the contents of the object pointed to are preserved up to the +* lesser of the new and old sizes. +* * Return: pointer to the allocated memory or %NULL in case of error */ void *krealloc_noprof (const void *p,size_t new_size,gfp_t flags)
Signature · 1 line
2.45.2
Andrew Morton
maintainerRe: [PATCH 2/2] mm: krealloc: clarify valid usage of __GFP_ZERO
Quoted 39 lines · On Tue, 30 Jul 2024 21:42:06 +0200 Danilo Krummrich <[email protected]> wrote:
Properly document that if
__GFP_ZEROlogic is requested, callers must ensure that, starting with the initial memory allocation, every subsequent call to this API for the same memory allocation is flagged with__GFP_ZERO. Otherwise, it is possible that__GFP_ZEROis not fully honored by this API....
include/linux/slab.h +8
-0@@ -733,6 +733,14 @@ static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t siz* @new_n: new number of elements to alloc * @new_size: new size of a single member of the array * @flags: the type of memory to allocate (see kmalloc) +* +* If __GFP_ZERO logic is requested, callers must ensure that, starting with the +* initial memory allocation, every subsequent call to this API for the same +* memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that +* __GFP_ZERO is not fully honored by this API. +* +* In any case, the contents of the object pointed to are preserved up to the +* lesser of the new and old sizes. */ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(void *p, size_t new_n,mm/slab_common.c +1
-2@@ -1301,11 +1301,17 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)* @new_size: how many bytes of memory are required. * @flags: the type of memory to allocate. * -* The contents of the object pointed to are preserved up to the -* lesser of the new and old sizes (__GFP_ZERO flag is effectively ignored). * If @p is %NULL, krealloc() behaves exactly like kmalloc(). If @new_size * is 0 and @p is not a %NULL pointer, the object pointed to is freed. * +* If __GFP_ZERO logic is requested, callers must ensure that, starting with the
mm/slab_common.c __do_krealloc(const void *p, size_t new_size, gfp_t flags) · line 1317
Quoted 10 lines
mm/slab_common.c +7
-0+* initial memory allocation, every subsequent call to this API for the same +* memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that +* __GFP_ZERO is not fully honored by this API. +* +* In any case, the contents of the object pointed to are preserved up to the +* lesser of the new and old sizes. +* * Return: pointer to the allocated memory or %NULL in case of error */ void *krealloc_noprof (const void *p,size_t new_size,gfp_t flags)
In both cases, we're saying "callers should do X". I think it would be better to say "this implementation does A, hence callers should do X". Tell people what's going on.
eg, "if krealloc is expanding an existing allocation, the newly-added memory will be uninitialized unless the caller used __GFP_ZERO". Or something like that.
I assume that if the caller actually touches the uninitialized memory, KASAN will warn?
Re: [PATCH 2/2] mm: krealloc: clarify valid usage of __GFP_ZERO
Quoted 47 lines · Andrew Morton
Quoted 45 lines · On Tue, 30 Jul 2024 21:42:06 +0200 Danilo Krummrich <[email protected]> wrote:
Properly document that if
__GFP_ZEROlogic is requested, callers must ensure that, starting with the initial memory allocation, every subsequent call to this API for the same memory allocation is flagged with__GFP_ZERO. Otherwise, it is possible that__GFP_ZEROis not fully honored by this API....
include/linux/slab.h +8
-0@@ -733,6 +733,14 @@ static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t siz* @new_n: new number of elements to alloc * @new_size: new size of a single member of the array * @flags: the type of memory to allocate (see kmalloc) +* +* If __GFP_ZERO logic is requested, callers must ensure that, starting with the +* initial memory allocation, every subsequent call to this API for the same +* memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that +* __GFP_ZERO is not fully honored by this API. +* +* In any case, the contents of the object pointed to are preserved up to the +* lesser of the new and old sizes. */ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(void *p, size_t new_n,mm/slab_common.c +7
-2@@ -1301,11 +1301,17 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)* @new_size: how many bytes of memory are required. * @flags: the type of memory to allocate. * -* The contents of the object pointed to are preserved up to the -* lesser of the new and old sizes (__GFP_ZERO flag is effectively ignored). * If @p is %NULL, krealloc() behaves exactly like kmalloc(). If @new_size * is 0 and @p is not a %NULL pointer, the object pointed to is freed. * +* If __GFP_ZERO logic is requested, callers must ensure that, starting with the +* initial memory allocation, every subsequent call to this API for the same +* memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that +* __GFP_ZERO is not fully honored by this API. +* +* In any case, the contents of the object pointed to are preserved up to the +* lesser of the new and old sizes.
Quoted 8 lines
mm/slab_common.c __do_krealloc(const void *p, size_t new_size, gfp_t flags) · line 1317
Quoted 4 lines
mm/slab_common.c +1
-0+* * Return: pointer to the allocated memory or %NULL in case of error */ void *krealloc_noprof (const void *p,size_t new_size,gfp_t flags)In both cases, we're saying "callers should do X". I think it would be better to say "this implementation does A, hence callers should do X". Tell people what's going on.
Sounds reasonable, I'll add an explanation here and in the fixup series for vrealloc() / kvrealloc().
Quoted 7 lines
eg, "if krealloc is expanding an existing allocation, the newly-added memory will be uninitialized unless the caller used
__GFP_ZERO". Or something like that.I assume that if the caller actually touches the uninitialized memory, KASAN will warn?
For the case that is fixed in patch 1 of this series, no. KASAN can't detect this.
As you say, the memory is just uninitialized (not poisoned), where it should have been zeroed instead.
Andrew Morton
maintainerQuoted 8 lines · On Tue, 30 Jul 2024 21:42:05 +0200 Danilo Krummrich <[email protected]> wrote:
As long as
krealloc()is called with__GFP_ZEROconsistently, starting with the initial memory allocation,__GFP_ZEROshould be fully honored.However, if for an existing allocation
krealloc()is called with a decreased size, it is not ensured that the spare portion the allocation is zeroed. Thus, ifkrealloc()is subsequently called with a larger size again,__GFP_ZEROcan't be fully honored, since we don't know the previous size, but only the bucket size.
Well that's bad.
Quoted 3 lines
Example:
buf = kzalloc(64, GFP_KERNEL);
If this was kmalloc()
Quoted 6 lines
memset(buf, 0xff, 64); buf = krealloc(buf, 48, GFP_KERNEL | __GFP_ZERO); /* After this call the last 16 bytes are still 0xff. */ buf = krealloc(buf, 64, GFP_KERNEL | __GFP_ZERO);
then this would expose uninitialized kernel memory to kernel code, with a risk that the kernel code will expose that to userspace, yes?
This does seem rather a trap, and I wonder whether krealloc() should just zero out any such data by default.
mm/slab_common.c __do_krealloc(const void *p, size_t new_size, gfp_t flags) · line 1279
Quoted 13 lines
Fix this, by explicitly setting spare memory to zero, when shrinking an allocation with
__GFP_ZEROflag set orinit_on_allocenabled.mm/slab_common.c +4
-0@@ -1273,6 +1273,13 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)/* If the object still fits, repoison it precisely. */ if (ks >= new_size) { +/* Zero out spare memory. */ +if (want_init_on_alloc (flags)) { +kasan_disable_current (); +memset ((void *)p + new_size,0 , ks - new_size);
Casting away the constness of *p'. This is just misleading everyone, really. It would be better to make argument p' have type "void *".
Quoted 4 lines
mm/slab_common.c +3
-0+kasan_enable_current (); + } + p =kasan_krealloc ((void *)p, new_size, flags);
Quoted 11 lines · Andrew Morton
Quoted 8 lines · On Tue, 30 Jul 2024 21:42:05 +0200 Danilo Krummrich <[email protected]> wrote:
As long as
krealloc()is called with__GFP_ZEROconsistently, starting with the initial memory allocation,__GFP_ZEROshould be fully honored.However, if for an existing allocation
krealloc()is called with a decreased size, it is not ensured that the spare portion the allocation is zeroed. Thus, ifkrealloc()is subsequently called with a larger size again,__GFP_ZEROcan't be fully honored, since we don't know the previous size, but only the bucket size.
Quoted 7 lines
Well that's bad.
Quoted 3 lines
Example:
buf = kzalloc(64, GFP_KERNEL);If this was
kmalloc()
Then already here we have unitialized kernel memory that a buggy user could expose, no?
Quoted 4 lines
Quoted 4 lines
memset(buf, 0xff, 64); buf = krealloc(buf, 48, GFP_KERNEL | __GFP_ZERO);
Quoted 8 lines
Quoted 2 lines
/* After this call the last 16 bytes are still 0xff. */ buf = krealloc(buf, 64, GFP_KERNEL | __GFP_ZERO);then this would expose uninitialized kernel memory to kernel code, with a risk that the kernel code will expose that to userspace, yes?
This does seem rather a trap, and I wonder whether
krealloc()should just zero out any such data by default.
So unless I'm missing how this differs from plain kmalloc(), relying on want_init_on_alloc() seems the right way how to opt-in harden against this potential exposure.
Quoted 22 lines
mm/slab_common.c __do_krealloc(const void *p, size_t new_size, gfp_t flags) · line 1279
Quoted 13 lines
Fix this, by explicitly setting spare memory to zero, when shrinking an allocation with
__GFP_ZEROflag set orinit_on_allocenabled.mm/slab_common.c +4
-0@@ -1273,6 +1273,13 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)/* If the object still fits, repoison it precisely. */ if (ks >= new_size) { +/* Zero out spare memory. */ +if (want_init_on_alloc (flags)) { +kasan_disable_current (); +memset ((void *)p + new_size,0 , ks - new_size);Casting away the constness of
*p'. This is just misleading everyone, really. It would be better to make argumentp' have type "void *".Quoted 4 lines
mm/slab_common.c +3
-0+kasan_enable_current (); + } + p =kasan_krealloc ((void *)p, new_size, flags);
Quoted 32 lines · Danilo Krummrich
As long as
krealloc()is called with__GFP_ZEROconsistently, starting with the initial memory allocation,__GFP_ZEROshould be fully honored.However, if for an existing allocation
krealloc()is called with a decreased size, it is not ensured that the spare portion the allocation is zeroed. Thus, ifkrealloc()is subsequently called with a larger size again,__GFP_ZEROcan't be fully honored, since we don't know the previous size, but only the bucket size.Example:
buf = kzalloc(64, GFP_KERNEL); memset(buf, 0xff, 64); buf = krealloc(buf, 48, GFP_KERNEL | __GFP_ZERO); /* After this call the last 16 bytes are still 0xff. */ buf = krealloc(buf, 64, GFP_KERNEL | __GFP_ZERO);Fix this, by explicitly setting spare memory to zero, when shrinking an allocation with
__GFP_ZEROflag set orinit_on_allocenabled.Signed-off-by Danilo Krummrich <[email protected]>--- mm/slab_common.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/slab_common.c +0
-0
mm/slab_common.c __do_krealloc(const void *p, size_t new_size, gfp_t flags) · line 1280
Quoted 8 lines
mm/slab_common.c +5
-0@@ -1273,6 +1273,13 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)/* If the object still fits, repoison it precisely. */ if (ks >= new_size) { +/* Zero out spare memory. */ +if (want_init_on_alloc (flags)) { +kasan_disable_current (); +memset ((void *)p + new_size,0 , ks - new_size); +kasan_enable_current ();
If we do kasan_krealloc() first, shouldn't the memset then be legal afterwards without the disable/enable dance?
Quoted 7 lines
mm/slab_common.c +2
-0+ } + p =kasan_krealloc ((void *)p, new_size, flags);return (void *)p; }base-commit:
7c3dd6d99f2df6a9d7944ee8505b195ba51c9b68
Quoted 34 lines · Vlastimil Babka
Quoted 33 lines · Danilo Krummrich
As long as
krealloc()is called with__GFP_ZEROconsistently, starting with the initial memory allocation,__GFP_ZEROshould be fully honored.However, if for an existing allocation
krealloc()is called with a decreased size, it is not ensured that the spare portion the allocation is zeroed. Thus, ifkrealloc()is subsequently called with a larger size again,__GFP_ZEROcan't be fully honored, since we don't know the previous size, but only the bucket size.Example:
buf = kzalloc(64, GFP_KERNEL); memset(buf, 0xff, 64); buf = krealloc(buf, 48, GFP_KERNEL | __GFP_ZERO); /* After this call the last 16 bytes are still 0xff. */ buf = krealloc(buf, 64, GFP_KERNEL | __GFP_ZERO);Fix this, by explicitly setting spare memory to zero, when shrinking an allocation with
__GFP_ZEROflag set orinit_on_allocenabled.Signed-off-by Danilo Krummrich <[email protected]>--- mm/slab_common.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/slab_common.c +0
-0@@ -1273,6 +1273,13 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)
Quoted 10 lines
mm/slab_common.c __do_krealloc(const void *p, size_t new_size, gfp_t flags) · line 1280
Quoted 7 lines
mm/slab_common.c +5
-0/* If the object still fits, repoison it precisely. */ if (ks >= new_size) { +/* Zero out spare memory. */ +if (want_init_on_alloc (flags)) { +kasan_disable_current (); +memset ((void *)p + new_size,0 , ks - new_size); +kasan_enable_current ();If we do
kasan_krealloc()first, shouldn't the memset then be legal afterwards without the disable/enable dance?
No, we always write into the poisoned area. The following tables show what we do in the particular case:
Shrink
------
new old
0 size size ks |----------|----------|----------| | keep | poison | <- poison |--------------------------------| | keep | zero | <- data
Poison and zero things between old size and ks is not necessary, but we don't know old size, hence we have do it between new size and ks.
Grow
----
old new
0 size size ks |----------|----------|----------| | unpoison | keep | <- poison |--------------------------------| | keep | zero | <- data
Zeroing between new_size and ks in not necessary in this case, since it must be zero already. But without knowing the old size we don't know whether we shrink and actually need to do something, or if we grow and don't need to do anything.
Analogously, we also unpoison things between 0 and old size for the same reason.
Quoted 9 lines
Quoted 8 lines
mm/slab_common.c +2
-0+ } + p =kasan_krealloc ((void *)p, new_size, flags);return (void *)p; }base-commit:
7c3dd6d99f2df6a9d7944ee8505b195ba51c9b68
Quoted 71 lines · Danilo Krummrich
Quoted 34 lines · Vlastimil Babka
Quoted 33 lines · Danilo Krummrich
As long as
krealloc()is called with__GFP_ZEROconsistently, starting with the initial memory allocation,__GFP_ZEROshould be fully honored.However, if for an existing allocation
krealloc()is called with a decreased size, it is not ensured that the spare portion the allocation is zeroed. Thus, ifkrealloc()is subsequently called with a larger size again,__GFP_ZEROcan't be fully honored, since we don't know the previous size, but only the bucket size.Example:
buf = kzalloc(64, GFP_KERNEL); memset(buf, 0xff, 64); buf = krealloc(buf, 48, GFP_KERNEL | __GFP_ZERO); /* After this call the last 16 bytes are still 0xff. */ buf = krealloc(buf, 64, GFP_KERNEL | __GFP_ZERO);Fix this, by explicitly setting spare memory to zero, when shrinking an allocation with
__GFP_ZEROflag set orinit_on_allocenabled.Signed-off-by Danilo Krummrich <[email protected]>--- mm/slab_common.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/slab_common.c +0
-0@@ -1273,6 +1273,13 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)Quoted 10 lines
mm/slab_common.c __do_krealloc(const void *p, size_t new_size, gfp_t flags) · line 1280
Quoted 7 lines
mm/slab_common.c +5
-0/* If the object still fits, repoison it precisely. */ if (ks >= new_size) { +/* Zero out spare memory. */ +if (want_init_on_alloc (flags)) { +kasan_disable_current (); +memset ((void *)p + new_size,0 , ks - new_size); +kasan_enable_current ();If we do
kasan_krealloc()first, shouldn't the memset then be legal afterwards without the disable/enable dance?No, we always write into the poisoned area. The following tables show what we do in the particular case:
Shrink
------
new old0 size size ks |----------|----------|----------| | keep | poison | <- poison |--------------------------------| | keep | zero | <- dataPoison and zero things between old size and ks is not necessary, but we don't know old size, hence we have do it between new size and ks.
Grow
----
old new0 size size ks |----------|----------|----------| | unpoison | keep | <- poison |--------------------------------| | keep | zero | <- data
Quoted 5 lines
Zeroing between
new_sizeand ks in not necessary in this case, since it must be zero already. But without knowing the old size we don't know whether we shrink and actually need to do something, or if we grow and don't need to do anything.Analogously, we also unpoison things between 0 and old size for the same reason.
Thanks, you're right!
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
mm/slab_common.c +2
-0+ } + p =kasan_krealloc ((void *)p, new_size, flags);return (void *)p; }base-commit:
7c3dd6d99f2df6a9d7944ee8505b195ba51c9b68
Quoted 19 lines · Danilo Krummrich
As long as
krealloc()is called with__GFP_ZEROconsistently, starting with the initial memory allocation,__GFP_ZEROshould be fully honored.However, if for an existing allocation
krealloc()is called with a decreased size, it is not ensured that the spare portion the allocation is zeroed. Thus, ifkrealloc()is subsequently called with a larger size again,__GFP_ZEROcan't be fully honored, since we don't know the previous size, but only the bucket size.Example:
buf = kzalloc(64, GFP_KERNEL); memset(buf, 0xff, 64); buf = krealloc(buf, 48, GFP_KERNEL | __GFP_ZERO); /* After this call the last 16 bytes are still 0xff. */ buf = krealloc(buf, 64, GFP_KERNEL | __GFP_ZERO);
Quoted 4 lines
Fix this, by explicitly setting spare memory to zero, when shrinking an allocation with
__GFP_ZEROflag set orinit_on_allocenabled.Signed-off-by Danilo Krummrich <[email protected]>
Quoted 24 lines
--- mm/slab_common.c | 7 +++++++ 1 file changed, 7 insertions(+)mm/slab_common.c +7
-0@@ -1273,6 +1273,13 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)/* If the object still fits, repoison it precisely. */ if (ks >= new_size) { +/* Zero out spare memory. */ +if (want_init_on_alloc (flags)) { +kasan_disable_current (); +memset ((void *)p + new_size,0 , ks - new_size); +kasan_enable_current (); + } + p =kasan_krealloc ((void *)p, new_size, flags);return (void *)p; }base-commit:
7c3dd6d99f2df6a9d7944ee8505b195ba51c9b68