mm: allow __GFP_RETRY_MAYFAIL in vmalloc
Subsystems (3)
Mikulas Patocka
maintainerPATCH 1/1 mm: allow __GFP_RETRY_MAYFAIL in vmalloc
The commit 07003531e03c8 ("mm/vmalloc: warn on invalid vmalloc gfp flags") breaks the device mapper VDO target. The VDO target calls vmalloc with __GFP_RETRY_MAYFAIL and this flag is not in the mask of allowed flags.
There is no reason why vmalloc couldn't support __GFP_RETRY_MAYFAIL, so let's add this flag to GFP_VMALLOC_SUPPORTED.
--- mm/vmalloc.c | 1 + 1 file changed, 1 insertion(+)
mm/vmalloc.c +1 -0
@@ -3928,6 +3928,7 @@ fail:*/ #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\ __GFP_NOFAIL | __GFP_ZERO | __GFP_NORETRY |\ + __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ GFP_USER | __GFP_NOLOCKDEP)
Quoted 5 lines · On Thu, 12 Feb 2026 17:33:30 +0100 (CET) Mikulas Patocka <[email protected]>…
The commit
07003531e03c8("mm/vmalloc: warn on invalid vmalloc gfp flags") breaks the device mapper VDO target. The VDO target calls vmalloc with__GFP_RETRY_MAYFAILand this flag is not in the mask of allowed flags.
Quoted 7 lines
There is no reason why vmalloc couldn't support
__GFP_RETRY_MAYFAIL, so let's add this flag toGFP_VMALLOC_SUPPORTED.Signed-off-by Mikulas Patocka <[email protected]>Reported-by Zdenek Kabelac <[email protected]>Fixes 07003531e03c ("mm/vmalloc: warn on invalid vmalloc gfp flags")Cc [email protected] # v6.19
Quoted 10 lines
--- mm/vmalloc.c | 1 + 1 file changed, 1 insertion(+)Index:
linux-2.6/mm/vmalloc.c
===================================================================
mm/vmalloc.c fail: · line 3933
Quoted 6 lines
mm/vmalloc.c +1
-0@@ -3928,6 +3928,7 @@ fail:*/ #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\ __GFP_NOFAIL | __GFP_ZERO | __GFP_NORETRY |\ + __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ GFP_USER | __GFP_NOLOCKDEP)
By the way, I think this list might better to be alphabetically sorted in future. Obviously just a trivial and random thought that never required for this patch.
Thanks,
SJ
[...]
Anshuman Khandual
Reviewed-byQuoted 22 lines · SeongJae Park
Quoted 5 lines · On Thu, 12 Feb 2026 17:33:30 +0100 (CET) Mikulas Patocka <[email protected]>…
The commit
07003531e03c8("mm/vmalloc: warn on invalid vmalloc gfp flags") breaks the device mapper VDO target. The VDO target calls vmalloc with__GFP_RETRY_MAYFAILand this flag is not in the mask of allowed flags.Quoted 7 lines
There is no reason why vmalloc couldn't support
__GFP_RETRY_MAYFAIL, so let's add this flag toGFP_VMALLOC_SUPPORTED.Signed-off-by Mikulas Patocka <[email protected]>Reported-by Zdenek Kabelac <[email protected]>Fixes 07003531e03c ("mm/vmalloc: warn on invalid vmalloc gfp flags")Cc [email protected] # v6.19Acked-by SeongJae Park <[email protected]>Quoted 5 lines
--- mm/vmalloc.c | 1 + 1 file changed, 1 insertion(+)
Quoted 13 lines
mm/vmalloc.c fail: · line 3933
Quoted 11 lines
Index:
linux-2.6/mm/vmalloc.c
===================================================================mm/vmalloc.c +1
-0@@ -3928,6 +3928,7 @@ fail:*/ #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\ __GFP_NOFAIL | __GFP_ZERO | __GFP_NORETRY |\ + __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ GFP_USER | __GFP_NOLOCKDEP)By the way, I think this list might better to be alphabetically sorted in
Agreed.
Quoted 2 lines
future. Obviously just a trivial and random thought that never required for this patch.
Quoted 7 lines · Mikulas Patocka
The commit
07003531e03c8("mm/vmalloc: warn on invalid vmalloc gfp flags") breaks the device mapper VDO target. The VDO target calls vmalloc with__GFP_RETRY_MAYFAILand this flag is not in the mask of allowed flags.There is no reason why vmalloc couldn't support
__GFP_RETRY_MAYFAIL, so let's add this flag toGFP_VMALLOC_SUPPORTED.
My only skepticism about this comes from the line in the vmalloc_node_range() doc:
"and %__GFP_RETRY_MAYFAIL are not supported."
I myself don't know why that may be. Could you elaborate on if/why the doc is wrong please?
Quoted 22 lines
Signed-off-by Mikulas Patocka <[email protected]>Reported-by Zdenek Kabelac <[email protected]>Fixes 07003531e03c ("mm/vmalloc: warn on invalid vmalloc gfp flags")Cc [email protected] # v6.19--- mm/vmalloc.c | 1 + 1 file changed, 1 insertion(+)Index:
linux-2.6/mm/vmalloc.c
===================================================================mm/vmalloc.c +1
-0@@ -3928,6 +3928,7 @@ fail:*/ #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\ __GFP_NOFAIL | __GFP_ZERO | __GFP_NORETRY |\ + __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ GFP_USER | __GFP_NOLOCKDEP)
Mikulas Patocka
maintainerQuoted 9 lines · Vishal Moola (Oracle)
Quoted 7 lines · Mikulas Patocka
The commit
07003531e03c8("mm/vmalloc: warn on invalid vmalloc gfp flags") breaks the device mapper VDO target. The VDO target calls vmalloc with__GFP_RETRY_MAYFAILand this flag is not in the mask of allowed flags.There is no reason why vmalloc couldn't support
__GFP_RETRY_MAYFAIL, so let's add this flag toGFP_VMALLOC_SUPPORTED.
Quoted 6 lines
My only skepticism about this comes from the line in the
vmalloc_node_range()doc:
"and %__GFP_RETRY_MAYFAIL are not supported."I myself don't know why that may be. Could you elaborate on if/why the doc is wrong please?
This statement was added by Michal Hocko in the commit b7d90e7a5ea8d64e668d5685925900d33d3884d5. Michal, could you explain why do you think that __GFP_RETRY_MAYFAIL is not supported?
The VDO module needs to allocate large amounts of memory and it doesn't want to trigger the OOM killer (which would kill some innocent task and wouldn't solve the out of memory condition at all), so I think that __GFP_RETRY_MAYFAIL is appropriate.
Mikulas
Quoted 23 lines
Quoted 22 lines
Signed-off-by Mikulas Patocka <[email protected]>Reported-by Zdenek Kabelac <[email protected]>Fixes 07003531e03c ("mm/vmalloc: warn on invalid vmalloc gfp flags")Cc [email protected] # v6.19--- mm/vmalloc.c | 1 + 1 file changed, 1 insertion(+)Index:
linux-2.6/mm/vmalloc.c
===================================================================mm/vmalloc.c +1
-0@@ -3928,6 +3928,7 @@ fail:*/ #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\ __GFP_NOFAIL | __GFP_ZERO | __GFP_NORETRY |\ + __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ GFP_USER | __GFP_NOLOCKDEP)
Uladzislau Rezki
maintainerQuoted 20 lines · Mikulas Patocka
Quoted 9 lines · Vishal Moola (Oracle)
Quoted 7 lines · Mikulas Patocka
The commit
07003531e03c8("mm/vmalloc: warn on invalid vmalloc gfp flags") breaks the device mapper VDO target. The VDO target calls vmalloc with__GFP_RETRY_MAYFAILand this flag is not in the mask of allowed flags.There is no reason why vmalloc couldn't support
__GFP_RETRY_MAYFAIL, so let's add this flag toGFP_VMALLOC_SUPPORTED.Quoted 6 lines
My only skepticism about this comes from the line in the
vmalloc_node_range()doc:
"and %__GFP_RETRY_MAYFAIL are not supported."I myself don't know why that may be. Could you elaborate on if/why the doc is wrong please?
Quoted 9 lines
This statement was added by Michal Hocko in the commit
b7d90e7a5ea8d64e668d5685925900d33d3884d5. Michal, could you explain why do you think that__GFP_RETRY_MAYFAILis not supported?The VDO module needs to allocate large amounts of memory and it doesn't want to trigger the OOM killer (which would kill some innocent task and wouldn't solve the out of memory condition at all), so I think that
__GFP_RETRY_MAYFAILis appropriate.
I do not see any problems using this with vmalloc from the first glance. KASAN path was hard-coded when it comes to GFP but it is not anymore.
--
Uladzislau Rezki
Quoted 17 lines · Mikulas Patocka
Quoted 13 lines · Vishal Moola (Oracle)
Quoted 7 lines · Mikulas Patocka
The commit
07003531e03c8("mm/vmalloc: warn on invalid vmalloc gfp flags") breaks the device mapper VDO target. The VDO target calls vmalloc with__GFP_RETRY_MAYFAILand this flag is not in the mask of allowed flags.There is no reason why vmalloc couldn't support
__GFP_RETRY_MAYFAIL, so let's add this flag toGFP_VMALLOC_SUPPORTED.My only skepticism about this comes from the line in the
vmalloc_node_range()doc:
"and %__GFP_RETRY_MAYFAIL are not supported."
Quoted 6 lines
Quoted 2 lines
I myself don't know why that may be. Could you elaborate on if/why the doc is wrong please?
This statement was added by Michal Hocko in the commit
b7d90e7a5ea8d64e668d5685925900d33d3884d5. Michal, could you explain why do you think that__GFP_RETRY_MAYFAILis not supported?
The problem with __GFP_RETRY_MAYFAIL is that it cannot be fully supported. While pages that back the allocation can be easily made aware of this failure mode there are page table allocations which are hardocded GFP_KERNEL and there is no sensible way to extend the API to change that (as we have learned several time over years).
Quoted 4 lines
The VDO module needs to allocate large amounts of memory and it doesn't want to trigger the OOM killer (which would kill some innocent task and wouldn't solve the out of memory condition at all), so I think that
__GFP_RETRY_MAYFAILis appropriate.
Understood. But as said the very page table allocation could be the trigger for the unwanted OOM. The same applies to __GFP_NORETRY unfortunately as well. vmalloc has just recently gained support of GFP_NOWAIT allocation mode, though. This will make the allocation failure much more likely though so I am not entirely sure this is a proper solution for your problem.
Signature · 2 lines
Michal Hocko SUSE Labs
Uladzislau Rezki
maintainerQuoted 36 lines · Michal Hocko
Quoted 17 lines · Mikulas Patocka
Quoted 13 lines · Vishal Moola (Oracle)
Quoted 7 lines · Mikulas Patocka
The commit
07003531e03c8("mm/vmalloc: warn on invalid vmalloc gfp flags") breaks the device mapper VDO target. The VDO target calls vmalloc with__GFP_RETRY_MAYFAILand this flag is not in the mask of allowed flags.There is no reason why vmalloc couldn't support
__GFP_RETRY_MAYFAIL, so let's add this flag toGFP_VMALLOC_SUPPORTED.My only skepticism about this comes from the line in the
vmalloc_node_range()doc:
"and %__GFP_RETRY_MAYFAIL are not supported."Quoted 6 lines
Quoted 2 lines
I myself don't know why that may be. Could you elaborate on if/why the doc is wrong please?
This statement was added by Michal Hocko in the commit
b7d90e7a5ea8d64e668d5685925900d33d3884d5. Michal, could you explain why do you think that__GFP_RETRY_MAYFAILis not supported?The problem with
__GFP_RETRY_MAYFAILis that it cannot be fully supported. While pages that back the allocation can be easily made aware of this failure mode there are page table allocations which are hardocdedGFP_KERNELand there is no sensible way to extend the API to change that (as we have learned several time over years).Quoted 4 lines
The VDO module needs to allocate large amounts of memory and it doesn't want to trigger the OOM killer (which would kill some innocent task and wouldn't solve the out of memory condition at all), so I think that
__GFP_RETRY_MAYFAILis appropriate.
Quoted 7 lines
Understood. But as said the very page table allocation could be the trigger for the unwanted OOM. The same applies to
__GFP_NORETRYunfortunately as well. vmalloc has just recently gained support ofGFP_NOWAITallocation mode, though. This will make the allocation failure much more likely though so I am not entirely sure this is a proper solution for your problem.
Yes, the page-table manipulation entries are hard-coded and it looks like it is the last path which is not wired properly with gfp-flags.
Since we grow PTEs and never release it might not be a big issue for the __GFP_RETRY_MAYFAIL usage. But it is still not valid in noted path.
--
Uladzislau Rezki
Quoted 44 lines · Uladzislau Rezki
Quoted 36 lines · Michal Hocko
Quoted 17 lines · Mikulas Patocka
Quoted 13 lines · Vishal Moola (Oracle)
Quoted 7 lines · Mikulas Patocka
The commit
07003531e03c8("mm/vmalloc: warn on invalid vmalloc gfp flags") breaks the device mapper VDO target. The VDO target calls vmalloc with__GFP_RETRY_MAYFAILand this flag is not in the mask of allowed flags.There is no reason why vmalloc couldn't support
__GFP_RETRY_MAYFAIL, so let's add this flag toGFP_VMALLOC_SUPPORTED.My only skepticism about this comes from the line in the
vmalloc_node_range()doc:
"and %__GFP_RETRY_MAYFAIL are not supported."Quoted 6 lines
Quoted 2 lines
I myself don't know why that may be. Could you elaborate on if/why the doc is wrong please?
This statement was added by Michal Hocko in the commit
b7d90e7a5ea8d64e668d5685925900d33d3884d5. Michal, could you explain why do you think that__GFP_RETRY_MAYFAILis not supported?The problem with
__GFP_RETRY_MAYFAILis that it cannot be fully supported. While pages that back the allocation can be easily made aware of this failure mode there are page table allocations which are hardocdedGFP_KERNELand there is no sensible way to extend the API to change that (as we have learned several time over years).Quoted 4 lines
The VDO module needs to allocate large amounts of memory and it doesn't want to trigger the OOM killer (which would kill some innocent task and wouldn't solve the out of memory condition at all), so I think that
__GFP_RETRY_MAYFAILis appropriate.Quoted 7 lines
Understood. But as said the very page table allocation could be the trigger for the unwanted OOM. The same applies to
__GFP_NORETRYunfortunately as well. vmalloc has just recently gained support ofGFP_NOWAITallocation mode, though. This will make the allocation failure much more likely though so I am not entirely sure this is a proper solution for your problem.
Quoted 5 lines
Yes, the page-table manipulation entries are hard-coded and it looks like it is the last path which is not wired properly with gfp-flags.
Since we grow PTEs and never release it might not be a big issue for the
__GFP_RETRY_MAYFAILusage. But it is still not valid in noted path.
One thing that we could do to improve __GFP_RETRY_MAYFAIL resp. __GFP_NORETRY is to use NOWAIT allocation semantic for page table allocations as those could be achieved by scoped allocation context. This could cause pre-mature failure after the whole bunch of memory has already been allocated for the backing pages but considering that page table allocations should be more and more rare over system runtime it might be just a reasonable workaround. WDYT?
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 7 lines · Michal Hocko
One thing that we could do to improve
__GFP_RETRY_MAYFAILresp.__GFP_NORETRYis to use NOWAIT allocation semantic for page table allocations as those could be achieved by scoped allocation context. This could cause pre-mature failure after the whole bunch of memory has already been allocated for the backing pages but considering that page table allocations should be more and more rare over system runtime it might be just a reasonable workaround. WDYT?
Why bother? __GFP_RETRY_MAYFAIL has pretty lose semantics. Trying too hard to allocate PTEs is not breaking the overall concept.
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Michal Hocko
One thing that we could do to improve
__GFP_RETRY_MAYFAILresp.__GFP_NORETRYis to use NOWAIT allocation semantic for page table allocations as those could be achieved by scoped allocation context. This could cause pre-mature failure after the whole bunch of memory has already been allocated for the backing pages but considering that page table allocations should be more and more rare over system runtime it might be just a reasonable workaround. WDYT?
Quoted 3 lines
Why bother?
__GFP_RETRY_MAYFAILhas pretty lose semantics. Trying too hard to allocate PTEs is not breaking the overall concept.
One thing __GFP_RETRY_MAYFAIL is very clear about is to not trigger the oom-killer which is not the case for GFP_KERNEL. There are users who explicitly use __GFP_RETRY_MAYFAIL to avoid oom-killer.
Mikulas, is that the reason you are using __GFP_RETRY_MAYFAIL in your use-case?
Quoted 7 lines · Shakeel Butt
Quoted 3 lines
Why bother?
__GFP_RETRY_MAYFAILhas pretty lose semantics. Trying too hard to allocate PTEs is not breaking the overall concept.One thing
__GFP_RETRY_MAYFAILis very clear about is to not trigger the oom-killer which is not the case forGFP_KERNEL. There are users who explicitly use__GFP_RETRY_MAYFAILto avoid oom-killer.
That is a good point. Compared to other options documenting that it might would still seem like the best option after just not using __GFP_RETRY_MAYFAIL with vmalloc.
Quoted 14 lines · Shakeel Butt
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Michal Hocko
One thing that we could do to improve
__GFP_RETRY_MAYFAILresp.__GFP_NORETRYis to use NOWAIT allocation semantic for page table allocations as those could be achieved by scoped allocation context. This could cause pre-mature failure after the whole bunch of memory has already been allocated for the backing pages but considering that page table allocations should be more and more rare over system runtime it might be just a reasonable workaround. WDYT?Quoted 3 lines
Why bother?
__GFP_RETRY_MAYFAILhas pretty lose semantics. Trying too hard to allocate PTEs is not breaking the overall concept.
Quoted 5 lines
One thing
__GFP_RETRY_MAYFAILis very clear about is to not trigger the oom-killer which is not the case forGFP_KERNEL. There are users who explicitly use__GFP_RETRY_MAYFAILto avoid oom-killer.Mikulas, is that the reason you are using
__GFP_RETRY_MAYFAILin your use-case?
yes https://lore.kernel.org/all/[email protected]/T/#u
Signature · 2 lines
Michal Hocko SUSE Labs
Mikulas Patocka
maintainerQuoted 14 lines · Shakeel Butt
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Michal Hocko
One thing that we could do to improve
__GFP_RETRY_MAYFAILresp.__GFP_NORETRYis to use NOWAIT allocation semantic for page table allocations as those could be achieved by scoped allocation context. This could cause pre-mature failure after the whole bunch of memory has already been allocated for the backing pages but considering that page table allocations should be more and more rare over system runtime it might be just a reasonable workaround. WDYT?Quoted 3 lines
Why bother?
__GFP_RETRY_MAYFAILhas pretty lose semantics. Trying too hard to allocate PTEs is not breaking the overall concept.
Quoted 5 lines
One thing
__GFP_RETRY_MAYFAILis very clear about is to not trigger the oom-killer which is not the case forGFP_KERNEL. There are users who explicitly use__GFP_RETRY_MAYFAILto avoid oom-killer.Mikulas, is that the reason you are using
__GFP_RETRY_MAYFAILin your use-case?
The VDO is deduplication storage driver - it needs to allocate large amounts of memory - for each disk block, it needs to store hash of the data in memory to be able to detect blocks with duplicate data.
If the VDO driver attempts to allocate more memory than what is available, we want the allocation to fail; we don't want to kill user processes.
Mikulas
Uladzislau Rezki
maintainerQuoted 51 lines · Michal Hocko
Quoted 44 lines · Uladzislau Rezki
Quoted 36 lines · Michal Hocko
Quoted 17 lines · Mikulas Patocka
Quoted 13 lines · Vishal Moola (Oracle)
Quoted 7 lines · Mikulas Patocka
The commit
07003531e03c8("mm/vmalloc: warn on invalid vmalloc gfp flags") breaks the device mapper VDO target. The VDO target calls vmalloc with__GFP_RETRY_MAYFAILand this flag is not in the mask of allowed flags.There is no reason why vmalloc couldn't support
__GFP_RETRY_MAYFAIL, so let's add this flag toGFP_VMALLOC_SUPPORTED.My only skepticism about this comes from the line in the
vmalloc_node_range()doc:
"and %__GFP_RETRY_MAYFAIL are not supported."Quoted 6 lines
Quoted 2 lines
I myself don't know why that may be. Could you elaborate on if/why the doc is wrong please?
This statement was added by Michal Hocko in the commit
b7d90e7a5ea8d64e668d5685925900d33d3884d5. Michal, could you explain why do you think that__GFP_RETRY_MAYFAILis not supported?The problem with
__GFP_RETRY_MAYFAILis that it cannot be fully supported. While pages that back the allocation can be easily made aware of this failure mode there are page table allocations which are hardocdedGFP_KERNELand there is no sensible way to extend the API to change that (as we have learned several time over years).Quoted 4 lines
The VDO module needs to allocate large amounts of memory and it doesn't want to trigger the OOM killer (which would kill some innocent task and wouldn't solve the out of memory condition at all), so I think that
__GFP_RETRY_MAYFAILis appropriate.Quoted 7 lines
Understood. But as said the very page table allocation could be the trigger for the unwanted OOM. The same applies to
__GFP_NORETRYunfortunately as well. vmalloc has just recently gained support ofGFP_NOWAITallocation mode, though. This will make the allocation failure much more likely though so I am not entirely sure this is a proper solution for your problem.Quoted 5 lines
Yes, the page-table manipulation entries are hard-coded and it looks like it is the last path which is not wired properly with gfp-flags.
Since we grow PTEs and never release it might not be a big issue for the
__GFP_RETRY_MAYFAILusage. But it is still not valid in noted path.
Quoted 8 lines
One thing that we could do to improve
__GFP_RETRY_MAYFAILresp.__GFP_NORETRYis to use NOWAIT allocation semantic for page table allocations as those could be achieved by scoped allocation context. This could cause pre-mature failure after the whole bunch of memory has already been allocated for the backing pages but considering that page table allocations should be more and more rare over system runtime it might be just a reasonable workaround. WDYT?
As far as i understand, Mikulas uses __GFP_RETRY_MAYFAIL with vmalloc for some time already. I have not seen any reports about that a PTE/xxx alloc path triggers OOM killer thus i tend to say that it is not easy to trigger.
I do not have a strong opinion about workaround you noted. Maybe Mikulas can switch to NOWAIT flag instead.
--
Uladzislau Rezki
Quoted 60 lines · Uladzislau Rezki
Quoted 51 lines · Michal Hocko
Quoted 44 lines · Uladzislau Rezki
Quoted 36 lines · Michal Hocko
Quoted 17 lines · Mikulas Patocka
Quoted 13 lines · Vishal Moola (Oracle)
Quoted 7 lines · Mikulas Patocka
The commit
07003531e03c8("mm/vmalloc: warn on invalid vmalloc gfp flags") breaks the device mapper VDO target. The VDO target calls vmalloc with__GFP_RETRY_MAYFAILand this flag is not in the mask of allowed flags.There is no reason why vmalloc couldn't support
__GFP_RETRY_MAYFAIL, so let's add this flag toGFP_VMALLOC_SUPPORTED.My only skepticism about this comes from the line in the
vmalloc_node_range()doc:
"and %__GFP_RETRY_MAYFAIL are not supported."Quoted 6 lines
Quoted 2 lines
I myself don't know why that may be. Could you elaborate on if/why the doc is wrong please?
This statement was added by Michal Hocko in the commit
b7d90e7a5ea8d64e668d5685925900d33d3884d5. Michal, could you explain why do you think that__GFP_RETRY_MAYFAILis not supported?The problem with
__GFP_RETRY_MAYFAILis that it cannot be fully supported. While pages that back the allocation can be easily made aware of this failure mode there are page table allocations which are hardocdedGFP_KERNELand there is no sensible way to extend the API to change that (as we have learned several time over years).Quoted 4 lines
The VDO module needs to allocate large amounts of memory and it doesn't want to trigger the OOM killer (which would kill some innocent task and wouldn't solve the out of memory condition at all), so I think that
__GFP_RETRY_MAYFAILis appropriate.Quoted 7 lines
Understood. But as said the very page table allocation could be the trigger for the unwanted OOM. The same applies to
__GFP_NORETRYunfortunately as well. vmalloc has just recently gained support ofGFP_NOWAITallocation mode, though. This will make the allocation failure much more likely though so I am not entirely sure this is a proper solution for your problem.Quoted 5 lines
Yes, the page-table manipulation entries are hard-coded and it looks like it is the last path which is not wired properly with gfp-flags.
Since we grow PTEs and never release it might not be a big issue for the
__GFP_RETRY_MAYFAILusage. But it is still not valid in noted path.Quoted 8 lines
One thing that we could do to improve
__GFP_RETRY_MAYFAILresp.__GFP_NORETRYis to use NOWAIT allocation semantic for page table allocations as those could be achieved by scoped allocation context. This could cause pre-mature failure after the whole bunch of memory has already been allocated for the backing pages but considering that page table allocations should be more and more rare over system runtime it might be just a reasonable workaround. WDYT?
Quoted 4 lines
As far as i understand, Mikulas uses
__GFP_RETRY_MAYFAILwith vmalloc for some time already. I have not seen any reports about that a PTE/xxx alloc path triggers OOM killer thus i tend to say that it is not easy to trigger.
Yes, triggering OOM killer is usually not all that easy. But rare OOMs are likely even worse because they are harder to test for. So I can see reasosning to avoid it as much as possible and rather fail early.
Quoted 2 lines
I do not have a strong opinion about workaround you noted. Maybe Mikulas can switch to NOWAIT flag instead.
using NOWAIT for the full vmalloc allocation would be just too easy to fail under moderate memory pressure.
The real question is whether we want to provide some sort of backoff early but not way too easily allocation semantic for vmalloc. If yes we need to get creative in the vmalloc internals rather than expect callers to be working around that on their side. History has proven that this just leads to tech. dept and more work later on.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 1 line · Michal Hocko
On Tue 24-02-26 16:38:01, Uladzislau Rezki wrote:
[...]
Quoted 11 lines
Quoted 2 lines
I do not have a strong opinion about workaround you noted. Maybe Mikulas can switch to NOWAIT flag instead.
using NOWAIT for the full vmalloc allocation would be just too easy to fail under moderate memory pressure.
The real question is whether we want to provide some sort of backoff early but not way too easily allocation semantic for vmalloc. If yes we need to get creative in the vmalloc internals rather than expect callers to be working around that on their side. History has proven that this just leads to tech. dept and more work later on.
Just to make sure we are on the same page I mean something like this
mm/vmalloc.c +11 -5
@@ -3798,6 +3798,8 @@ static void defer_vm_area_cleanup(struct vm_struct *area)* non-blocking (no __GFP_DIRECT_RECLAIM) - memalloc_noreclaim_save() * GFP_NOFS - memalloc_nofs_save() * GFP_NOIO - memalloc_noio_save() +* __GFP_RETRY_MAYFAIL, __GFP_NORETRY - memalloc_noreclaim_save() to prevent +* OOMs * * Returns a flag cookie to pair with restore. */
@@ -3806,7 +3808,7 @@ memalloc_apply_gfp_scope(gfp_t gfp_mask) {unsigned int flags =0 ; -if (!gfpflags_allow_blocking (gfp_mask)) +if (!gfpflags_allow_blocking (gfp_mask) || (gfp_mask & (__GFP_RETRY_MAYFAIL|__GFP_NORETRY))) flags =memalloc_noreclaim_save ();else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO) flags =memalloc_nofs_save ();
@@ -3940,7 +3942,8 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,* GFP_KERNEL_ACCOUNT. Xfs uses __GFP_NOLOCKDEP. */ #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\ - __GFP_NOFAIL | __GFP_ZERO | __GFP_NORETRY |\ + __GFP_NOFAIL | __GFP_ZERO | |\ + __GFP_NORETRY | __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ GFP_USER | __GFP_NOLOCKDEP)
@@ -3971,12 +3974,15 @@ static gfp_t vmalloc_fix_flags(gfp_t flags)* virtual range with protection @prot. * * Supported GFP classes: %GFP_KERNEL, %GFP_ATOMIC, %GFP_NOWAIT, -* %GFP_NOFS and %GFP_NOIO. Zone modifiers are not supported. +* %__GFP_RETRY_MAYFAIL, %__GFP_NORETRY, %GFP_NOFS and %GFP_NOIO. +* Zone modifiers are not supported. * Please note %GFP_ATOMIC and %GFP_NOWAIT are supported only * by __vmalloc(). * -* Retry modifiers: only %__GFP_NOFAIL is supported; %__GFP_NORETRY -* and %__GFP_RETRY_MAYFAIL are not supported. +* Retry modifiers: only %__GFP_NOFAIL is fully supported; +* %__GFP_NORETRY and %__GFP_RETRY_MAYFAIL are supported with limitation, +* i.e. page tables are allocated with NOWAIT semantic so they might fail +* under moderate memory pressure. * * %__GFP_NOWARN can be used to suppress failure messages. *
Signature · 2 lines
Michal Hocko SUSE Labs
Uladzislau Rezki
maintainerQuoted 58 lines · Michal Hocko
Quoted 1 line · Michal Hocko
On Tue 24-02-26 16:38:01, Uladzislau Rezki wrote:
[...]
Quoted 11 lines
Quoted 2 lines
I do not have a strong opinion about workaround you noted. Maybe Mikulas can switch to NOWAIT flag instead.
using NOWAIT for the full vmalloc allocation would be just too easy to fail under moderate memory pressure.
The real question is whether we want to provide some sort of backoff early but not way too easily allocation semantic for vmalloc. If yes we need to get creative in the vmalloc internals rather than expect callers to be working around that on their side. History has proven that this just leads to tech. dept and more work later on.
Just to make sure we are on the same page I mean something like this
mm/vmalloc.c +7
-3@@ -3798,6 +3798,8 @@ static void defer_vm_area_cleanup(struct vm_struct *area)* non-blocking (no __GFP_DIRECT_RECLAIM) - memalloc_noreclaim_save() * GFP_NOFS - memalloc_nofs_save() * GFP_NOIO - memalloc_noio_save() +* __GFP_RETRY_MAYFAIL, __GFP_NORETRY - memalloc_noreclaim_save() to prevent +* OOMs * * Returns a flag cookie to pair with restore. */ @@ -3806,7 +3808,7 @@ memalloc_apply_gfp_scope(gfp_t gfp_mask) {unsigned int flags =0 ; -if (!gfpflags_allow_blocking (gfp_mask)) +if (!gfpflags_allow_blocking (gfp_mask) || (gfp_mask & (__GFP_RETRY_MAYFAIL|__GFP_NORETRY))) flags =memalloc_noreclaim_save ();else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO) flags =memalloc_nofs_save ();@@ -3940,7 +3942,8 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,* GFP_KERNEL_ACCOUNT. Xfs uses __GFP_NOLOCKDEP. */ #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\ - __GFP_NOFAIL | __GFP_ZERO | __GFP_NORETRY |\ + __GFP_NOFAIL | __GFP_ZERO | |\ + __GFP_NORETRY | __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ GFP_USER | __GFP_NOLOCKDEP)@@ -3971,12 +3974,15 @@ static gfp_t vmalloc_fix_flags(gfp_t flags)* virtual range with protection @prot. * * Supported GFP classes: %GFP_KERNEL, %GFP_ATOMIC, %GFP_NOWAIT, -* %GFP_NOFS and %GFP_NOIO. Zone modifiers are not supported. +* %__GFP_RETRY_MAYFAIL, %__GFP_NORETRY, %GFP_NOFS and %GFP_NOIO. +* Zone modifiers are not supported. * Please note %GFP_ATOMIC and %GFP_NOWAIT are supported only * by __vmalloc().
mm/vmalloc.c static gfp_t vmalloc_fix_flags(gfp_t flags) · line 3988
Quoted 10 lines
mm/vmalloc.c +4
-2* -* Retry modifiers: only %__GFP_NOFAIL is supported; %__GFP_NORETRY -* and %__GFP_RETRY_MAYFAIL are not supported. +* Retry modifiers: only %__GFP_NOFAIL is fully supported; +* %__GFP_NORETRY and %__GFP_RETRY_MAYFAIL are supported with limitation, +* i.e. page tables are allocated with NOWAIT semantic so they might fail +* under moderate memory pressure. * * %__GFP_NOWARN can be used to suppress failure messages. *
Yep, i got your intention correctly. It would eliminate the problem with page tables allocations :)
--
Uladzislau Rezki
Uladzislau Rezki
maintainerQuoted 63 lines · Uladzislau Rezki
Quoted 62 lines · Michal Hocko
Quoted 1 line · Michal Hocko
On Tue 24-02-26 16:38:01, Uladzislau Rezki wrote:
[...]
Quoted 11 lines
Quoted 2 lines
I do not have a strong opinion about workaround you noted. Maybe Mikulas can switch to NOWAIT flag instead.
using NOWAIT for the full vmalloc allocation would be just too easy to fail under moderate memory pressure.
The real question is whether we want to provide some sort of backoff early but not way too easily allocation semantic for vmalloc. If yes we need to get creative in the vmalloc internals rather than expect callers to be working around that on their side. History has proven that this just leads to tech. dept and more work later on.
Just to make sure we are on the same page I mean something like this
mm/vmalloc.c +9
-5@@ -3798,6 +3798,8 @@ static void defer_vm_area_cleanup(struct vm_struct *area)* non-blocking (no __GFP_DIRECT_RECLAIM) - memalloc_noreclaim_save() * GFP_NOFS - memalloc_nofs_save() * GFP_NOIO - memalloc_noio_save() +* __GFP_RETRY_MAYFAIL, __GFP_NORETRY - memalloc_noreclaim_save() to prevent +* OOMs * * Returns a flag cookie to pair with restore. */ @@ -3806,7 +3808,7 @@ memalloc_apply_gfp_scope(gfp_t gfp_mask) {unsigned int flags =0 ; -if (!gfpflags_allow_blocking (gfp_mask)) +if (!gfpflags_allow_blocking (gfp_mask) || (gfp_mask & (__GFP_RETRY_MAYFAIL|__GFP_NORETRY))) flags =memalloc_noreclaim_save ();else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO) flags =memalloc_nofs_save ();@@ -3940,7 +3942,8 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,* GFP_KERNEL_ACCOUNT. Xfs uses __GFP_NOLOCKDEP. */ #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\ - __GFP_NOFAIL | __GFP_ZERO | __GFP_NORETRY |\ + __GFP_NOFAIL | __GFP_ZERO | |\ + __GFP_NORETRY | __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ GFP_USER | __GFP_NOLOCKDEP)@@ -3971,12 +3974,15 @@ static gfp_t vmalloc_fix_flags(gfp_t flags)* virtual range with protection @prot. * * Supported GFP classes: %GFP_KERNEL, %GFP_ATOMIC, %GFP_NOWAIT, -* %GFP_NOFS and %GFP_NOIO. Zone modifiers are not supported. +* %__GFP_RETRY_MAYFAIL, %__GFP_NORETRY, %GFP_NOFS and %GFP_NOIO. +* Zone modifiers are not supported. * Please note %GFP_ATOMIC and %GFP_NOWAIT are supported only * by __vmalloc(). * -* Retry modifiers: only %__GFP_NOFAIL is supported; %__GFP_NORETRY -* and %__GFP_RETRY_MAYFAIL are not supported. +* Retry modifiers: only %__GFP_NOFAIL is fully supported; +* %__GFP_NORETRY and %__GFP_RETRY_MAYFAIL are supported with limitation,
Quoted 9 lines
mm/vmalloc.c static gfp_t vmalloc_fix_flags(gfp_t flags) · line 3988
Quoted 6 lines
mm/vmalloc.c +2
-0+* i.e. page tables are allocated with NOWAIT semantic so they might fail +* under moderate memory pressure. * * %__GFP_NOWARN can be used to suppress failure messages. * Yep, i got your intention correctly. It would eliminate the problem with page tables allocations :)
If no objection, we can go the way which was proposed by Michal and support __GFP_RETRY_MAYFAIL flag.
Mikulas is that possible for you to resend the patch including Michal changes(unless Michal wants to send the patch on his own)?
--
Uladzislau Rezki
Feel free to use this patch. But please note that I do not have any user so I couldn't test it. So make sure this is tested before posted for official inclusion.
---
From ee94ade23655dac7e239c1dfd2b04c0db3d10298 Mon Sep 17 00:00:00 2001
From: Michal Hocko <[email protected]>
Date: Wed, 25 Feb 2026 12:38:28 +0100
Subject: [PATCH] vmalloc: support __GFP_RETRY_MAYFAIL and __GFP_NORETRY
__GFP_RETRY_MAYFAIL and __GFP_NORETRY haven't been supported so far because their semantic (i.e. to not trigger OOM killer) is not possible with the existing vmalloc page table allocation which is allowing for the OOM killer.
There are usecases for these modifiers when a large allocation request should rather fail than trigger OOM killer which wouldn't be able to handle the situation anyway [1].
While we cannot change existing page table allocation code easily we can piggy back on scoped NOWAIT allocation for them that we already have in place. The rationale is that the bulk of the consumed memory is sitting in pages backing the vmalloc allocation. Page tables are only participating a tiny fraction. Moreover page tables for virtually allocated areas are never reclaimed so the longer the system runs to less likely they are. So it makes sense to allow an approximation of __GFP_RETRY_MAYFAIL and __GFP_NORETRY even if the page table allocation part is much weaker. This doesn't break the failure mode while it allows for the no OOM semantic.
[1] https://lore.kernel.org/all/[email protected]/T/#u
--- mm/vmalloc.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
mm/vmalloc.c +11 -5
@@ -3798,6 +3798,8 @@ static void defer_vm_area_cleanup(struct vm_struct *area)* non-blocking (no __GFP_DIRECT_RECLAIM) - memalloc_noreclaim_save() * GFP_NOFS - memalloc_nofs_save() * GFP_NOIO - memalloc_noio_save() +* __GFP_RETRY_MAYFAIL, __GFP_NORETRY - memalloc_noreclaim_save() to prevent +* OOMs * * Returns a flag cookie to pair with restore. */
@@ -3806,7 +3808,7 @@ memalloc_apply_gfp_scope(gfp_t gfp_mask) {unsigned int flags =0 ; -if (!gfpflags_allow_blocking (gfp_mask)) +if (!gfpflags_allow_blocking (gfp_mask) || (gfp_mask & (__GFP_RETRY_MAYFAIL|__GFP_NORETRY))) flags =memalloc_noreclaim_save ();else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO) flags =memalloc_nofs_save ();
@@ -3940,7 +3942,8 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,* GFP_KERNEL_ACCOUNT. Xfs uses __GFP_NOLOCKDEP. */ #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\ - __GFP_NOFAIL | __GFP_ZERO | __GFP_NORETRY |\ + __GFP_NOFAIL | __GFP_ZERO | |\ + __GFP_NORETRY | __GFP_RETRY_MAYFAIL |\ GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\ GFP_USER | __GFP_NOLOCKDEP)
@@ -3971,12 +3974,15 @@ static gfp_t vmalloc_fix_flags(gfp_t flags)* virtual range with protection @prot. * * Supported GFP classes: %GFP_KERNEL, %GFP_ATOMIC, %GFP_NOWAIT, -* %GFP_NOFS and %GFP_NOIO. Zone modifiers are not supported. +* %__GFP_RETRY_MAYFAIL, %__GFP_NORETRY, %GFP_NOFS and %GFP_NOIO. +* Zone modifiers are not supported. * Please note %GFP_ATOMIC and %GFP_NOWAIT are supported only * by __vmalloc(). * -* Retry modifiers: only %__GFP_NOFAIL is supported; %__GFP_NORETRY -* and %__GFP_RETRY_MAYFAIL are not supported. +* Retry modifiers: only %__GFP_NOFAIL is fully supported; +* %__GFP_NORETRY and %__GFP_RETRY_MAYFAIL are supported with limitation, +* i.e. page tables are allocated with NOWAIT semantic so they might fail +* under moderate memory pressure. * * %__GFP_NOWARN can be used to suppress failure messages. *
Signature · 5 lines
2.53.0 -- Michal Hocko SUSE Labs
Uladzislau Rezki
maintainerQuoted 5 lines
Feel free to use this patch. But please note that I do not have any user so I couldn't test it. So make sure this is tested before posted for official inclusion.
Thank you. I will properly test it and apply my fixes if needed.
--
Uladzislau Rezki