dm-region-hash: fix strange usage of mempool_alloc.
PATCH 1/1 dm-region-hash: fix strange usage of mempool_alloc.
mempool_alloc() should only be called with GFP_ATOMIC when it is not safe to wait. Passing __GFP_NOFAIL to kmalloc() says that it is safe to wait indefinitely. So this code is inconsistent.
Clearly it is OK to wait indefinitely in this code, and mempool_alloc() is able to do that. So just use mempool_alloc, and allow it to sleep. If no memory is available it will wait for something to be returned to the pool, and will retry a normal allocation regularly.
--- drivers/md/dm-region-hash.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/md/dm-region-hash.c +1 -3
@@ -287,9 +287,7 @@ static struct dm_region *__rh_alloc(struct dm_region_hash *rh, region_t region) {struct dm_region *reg, *nreg; - nreg =mempool_alloc (rh->region_pool,GFP_ATOMIC ); -if (unlikely (!nreg)) - nreg =kmalloc (sizeof (*nreg),GFP_NOIO | __GFP_NOFAIL); + nreg =mempool_alloc (rh->region_pool,GFP_NOIO ); nreg->state = rh->log->type->in_sync (rh->log, region,1 ) ?DM_RH_CLEAN :DM_RH_NOSYNC ;
Signature · 1 line
2.12.2
Mikulas Patocka
maintainerQuoted 10 lines · NeilBrown
mempool_alloc()should only be called withGFP_ATOMICwhen it is not safe to wait. Passing__GFP_NOFAILtokmalloc()says that it is safe to wait indefinitely. So this code is inconsistent.Clearly it is OK to wait indefinitely in this code, and
mempool_alloc()is able to do that. So just usemempool_alloc, and allow it to sleep. If no memory is available it will wait for something to be returned to the pool, and will retry a normal allocation regularly.
The region hash code is buggy anyway, because it may allocate more entries than the size of the pool and not give them back.
That kmalloc was introduced in the commit c06aad854 to work around a deadlock due to incorrect mempool usage.
Your patch slightly increases the probability of the deadlock because mempool_alloc does all allocations with __GFP_NOMEMALLOC, so it uses higher limit than kmalloc(GFP_NOIO).
Mikulas
Quoted 24 lines
Signed-off-by NeilBrown <[email protected]>--- drivers/md/dm-region-hash.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)drivers/md/dm-region-hash.c +1
-3@@ -287,9 +287,7 @@ static struct dm_region *__rh_alloc(struct dm_region_hash *rh, region_t region) {struct dm_region *reg, *nreg; - nreg =mempool_alloc (rh->region_pool,GFP_ATOMIC ); -if (unlikely (!nreg)) - nreg =kmalloc (sizeof (*nreg),GFP_NOIO | __GFP_NOFAIL); + nreg =mempool_alloc (rh->region_pool,GFP_NOIO ); nreg->state = rh->log->type->in_sync (rh->log, region,1 ) ?DM_RH_CLEAN :DM_RH_NOSYNC ;Signature · 1 line
2.12.2
Quoted 16 lines · Mikulas Patocka
Quoted 10 lines · NeilBrown
mempool_alloc()should only be called withGFP_ATOMICwhen it is not safe to wait. Passing__GFP_NOFAILtokmalloc()says that it is safe to wait indefinitely. So this code is inconsistent.Clearly it is OK to wait indefinitely in this code, and
mempool_alloc()is able to do that. So just usemempool_alloc, and allow it to sleep. If no memory is available it will wait for something to be returned to the pool, and will retry a normal allocation regularly.The region hash code is buggy anyway, because it may allocate more entries than the size of the pool and not give them back.
Quoted 7 lines
That kmalloc was introduced in the commit
c06aad854to work around a deadlock due to incorrect mempool usage.Your patch slightly increases the probability of the deadlock because
mempool_allocdoes all allocations with__GFP_NOMEMALLOC, so it uses higher limit thankmalloc(GFP_NOIO).
Thanks for the review.
I had a look at how the allocation 'dm_region' objects are used, and it would take a bit of work to make it really safe.
My guess is __rh_find() should be allowed to fail, and the various callers need to handle failure.
For example, dm_rh_inc_pending() would be given a second bio_list, and would move any bios for which rh_inc() fails, onto that list. Then do_writes() would merge that list back into ms->writes. That way do_mirror() would not block indefinitely and forward progress could be assured ... maybe.
It would take more work than I'm able to give at the moment, so I'm happy to just drop this patch.
Thanks,
NeilBrown
Mikulas Patocka
maintainerQuoted 40 lines · NeilBrown
Quoted 16 lines · Mikulas Patocka
Quoted 10 lines · NeilBrown
mempool_alloc()should only be called withGFP_ATOMICwhen it is not safe to wait. Passing__GFP_NOFAILtokmalloc()says that it is safe to wait indefinitely. So this code is inconsistent.Clearly it is OK to wait indefinitely in this code, and
mempool_alloc()is able to do that. So just usemempool_alloc, and allow it to sleep. If no memory is available it will wait for something to be returned to the pool, and will retry a normal allocation regularly.The region hash code is buggy anyway, because it may allocate more entries than the size of the pool and not give them back.
Quoted 7 lines
That kmalloc was introduced in the commit
c06aad854to work around a deadlock due to incorrect mempool usage.Your patch slightly increases the probability of the deadlock because
mempool_allocdoes all allocations with__GFP_NOMEMALLOC, so it uses higher limit thankmalloc(GFP_NOIO).Thanks for the review.
I had a look at how the allocation '
dm_region' objects are used, and it would take a bit of work to make it really safe.
My guess is__rh_find()should be allowed to fail, and the various callers need to handle failure.
For example,dm_rh_inc_pending()would be given a secondbio_list, and would move any bios for whichrh_inc()fails, onto that list. Thendo_writes()would merge that list back intoms->writes. That waydo_mirror()would not block indefinitely and forward progress could be assured ... maybe.
It would take more work than I'm able to give at the moment, so I'm happy to just drop this patch.
Quoted 2 lines
Thanks,
NeilBrown
I think that the only way how to fix this would be to preallocate the all the regions when the target is created.
But, with the default region size 512kiB, it would cause high memory consumption (approximatelly 1GB of RAM for 20TB device).
Mikulas
Quoted 14 lines · Mikulas Patocka
Quoted 13 lines · NeilBrown
I had a look at how the allocation '
dm_region' objects are used, and it would take a bit of work to make it really safe.
My guess is__rh_find()should be allowed to fail, and the various callers need to handle failure.
For example,dm_rh_inc_pending()would be given a secondbio_list, and would move any bios for whichrh_inc()fails, onto that list. Thendo_writes()would merge that list back intoms->writes. That waydo_mirror()would not block indefinitely and forward progress could be assured ... maybe.
It would take more work than I'm able to give at the moment, so I'm happy to just drop this patch.
Quoted 8 lines
Quoted 2 lines
Thanks,
NeilBrownI think that the only way how to fix this would be to preallocate the all the regions when the target is created.
But, with the default region size 512kiB, it would cause high memory consumption (approximatelly 1GB of RAM for 20TB device).
Two reflections:
1/ This is close to what md/bitmap does.
It actually uses a 2-level array for the 'pending' field from dm_region, combined with something similar to 'state'. The top level is allocated when the device is created. Entries in this table are either
- pointers to a second-level array for 2048 regions
- entries for 2 giant regions, 1024 times the normal size.
So if we cannot allocate a page when we need that second level,
we just use an enormous region and so risk resync taking a bit longer if there is a crash.
2/ Even though md does pre-alloc to a degree, I'm not convinced that it
is necessary.
We only need a region to be recorded when it is actively being written to, or when it is being recovered.
We could, in theory, have just one region that is written to and one region that is being recovered. If a writes request arrives for a different region it blocks until the current region has no active requests. Then that region is forgotten and the new region activated, and the new write allowed to proceed. Obviously this would be horribly slow, but it should be deadlock-free.
Using a mempool instead of a single region would then allow multiple regions to be active in parallel, which would improve throughput without affecting the deadlock status.
Maybe I'll try to code it and see what happens ... maybe not.
NeilBrown