Skip to contentSkip to conversation

Remove blkg node after destroying blkg

PATCH In review Tao Su 3 participants 6 messages Apr 25, 2023 – Apr 26, 2023

Subsystems (2)
lore
Share & tools
lore

Tao Su

author
#0 lore raw

PATCH 1/1 Remove blkg node after destroying blkg

Kernel hang when poweroff or reboot, due to infinite restart in function blkg_destroy_all. It will goto restart label when a batch of blkgs are destroyed, but not remove blkg node in blkg_list. So the blkg_list is same in every 'restart' and result in kernel hang.

By adding list_del to remove blkg node after destroying, can solve this kernel hang issue and satisfy the previous will to 'restart'.

---
 block/blk-cgroup.c | 1 +
 1 file changed, 1 insertion(+)
block/blk-cgroup.c +1 -0
index bd50b55bdb61..960eb538a704 100644
@@ -530,6 +530,7 @@ static void blkg_destroy_all(struct gendisk *disk)
 
 		spin_lock(&blkcg->lock);
 		blkg_destroy(blkg);
+		list_del(&blkg->q_node);
 		spin_unlock(&blkcg->lock);
 
 		/*
Signature · 1 line
2.34.1
#1 lore raw

Hi,

Quoted 21 lines · Tao Su

Kernel hang when poweroff or reboot, due to infinite restart in function blkg_destroy_all. It will goto restart label when a batch of blkgs are destroyed, but not remove blkg node in blkg_list. So the blkg_list is same in every 'restart' and result in kernel hang.

By adding list_del to remove blkg node after destroying, can solve this kernel hang issue and satisfy the previous will to 'restart'.

---
  block/blk-cgroup.c | 1 +
  1 file changed, 1 insertion(+)
block/blk-cgroup.c +0 -0
index bd50b55bdb61..960eb538a704 100644

block/blk-cgroup.c static void blkg_destroy_all(struct gendisk *disk) · line 533

Quoted 4 lines
block/blk-cgroup.c +1 -0
@@ -530,6 +530,7 @@ static void blkg_destroy_all(struct gendisk *disk)
 
  		spin_lock(&blkcg->lock);
  		blkg_destroy(blkg);
+		list_del(&blkg->q_node);

blkg should stay on the queue list until blkg_free_workfn(), otherwise parent blkg can be freed before child, which will cause some known issue.

I think this hung happens when total blkg is greater than BLKG_DESTROY_BATCH_SIZE, right?

Can you try if following patch fix your problem?

index 1c1ebeb51003..0ecb4cce8af2 100644

block/blk-cgroup.c +3 -0
@@ -527,6 +527,9 @@ static void blkg_destroy_all(struct gendisk *disk)
        list_for_each_entry_safe(blkg, n, &q->blkg_list, q_node) {
                struct blkcg *blkcg = blkg->blkcg;
 
+               if (hlist_unhashed(&blkg->blkcg_node))
+                       continue;
+
                spin_lock(&blkcg->lock);
                blkg_destroy(blkg);
                spin_unlock(&blkcg->lock);
Quoted 4 lines
block/blk-cgroup.c +0 -0
  		spin_unlock(&blkcg->lock);
 
  		/*
author
#2 lore raw
Quoted 19 lines · Yu Kuai

Hi,

Quoted 16 lines · Tao Su

Kernel hang when poweroff or reboot, due to infinite restart in function blkg_destroy_all. It will goto restart label when a batch of blkgs are destroyed, but not remove blkg node in blkg_list. So the blkg_list is same in every 'restart' and result in kernel hang.

By adding list_del to remove blkg node after destroying, can solve this kernel hang issue and satisfy the previous will to 'restart'.

---
  block/blk-cgroup.c | 1 +
  1 file changed, 1 insertion(+)
Quoted 12 lines

block/blk-cgroup.c static void blkg_destroy_all(struct gendisk *disk) · line 532

Quoted 8 lines
block/blk-cgroup.c +1 -0
index bd50b55bdb61..960eb538a704 100644
@@ -530,6 +530,7 @@ static void blkg_destroy_all(struct gendisk *disk)
  		spin_lock(&blkcg->lock);
  		blkg_destroy(blkg);
+		list_del(&blkg->q_node);

blkg should stay on the queue list until blkg_free_workfn(), otherwise parent blkg can be freed before child, which will cause some known issue.

Yes, directly removing blkg node is not appropriate, which I noticed some comments in blkg_destroy(), thanks for pointing out this issue.

Quoted 3 lines

I think this hung happens when total blkg is greater than BLKG_DESTROY_BATCH_SIZE, right?

Yes, you are right.

Quoted 2 lines

Can you try if following patch fix your problem?

This patch can also fix my problem, and indeed is a more secure way.

Thanks,
Tao

Quoted 18 lines

index 1c1ebeb51003..0ecb4cce8af2 100644

block/blk-cgroup.c +3 -0
@@ -527,6 +527,9 @@ static void blkg_destroy_all(struct gendisk *disk)
        list_for_each_entry_safe(blkg, n, &q->blkg_list, q_node) {
                struct blkcg *blkcg = blkg->blkcg;
 
+               if (hlist_unhashed(&blkg->blkcg_node))
+                       continue;
+
                spin_lock(&blkcg->lock);
                blkg_destroy(blkg);
                spin_unlock(&blkcg->lock);
Quoted 4 lines
block/blk-cgroup.c +0 -0
  		spin_unlock(&blkcg->lock);
  		/*
 
#3 lore raw

Hi,

Quoted 40 lines · Tao Su
Quoted 19 lines · Yu Kuai

Hi,

Quoted 16 lines · Tao Su

Kernel hang when poweroff or reboot, due to infinite restart in function blkg_destroy_all. It will goto restart label when a batch of blkgs are destroyed, but not remove blkg node in blkg_list. So the blkg_list is same in every 'restart' and result in kernel hang.

By adding list_del to remove blkg node after destroying, can solve this kernel hang issue and satisfy the previous will to 'restart'.

---
   block/blk-cgroup.c | 1 +
   1 file changed, 1 insertion(+)
Quoted 12 lines

block/blk-cgroup.c static void blkg_destroy_all(struct gendisk *disk) · line 532

Quoted 8 lines
block/blk-cgroup.c +1 -0
index bd50b55bdb61..960eb538a704 100644
@@ -530,6 +530,7 @@ static void blkg_destroy_all(struct gendisk *disk)
   		spin_lock(&blkcg->lock);
   		blkg_destroy(blkg);
+		list_del(&blkg->q_node);

blkg should stay on the queue list until blkg_free_workfn(), otherwise parent blkg can be freed before child, which will cause some known issue.

Yes, directly removing blkg node is not appropriate, which I noticed some comments in blkg_destroy(), thanks for pointing out this issue.

Quoted 3 lines

I think this hung happens when total blkg is greater than BLKG_DESTROY_BATCH_SIZE, right?

Quoted 6 lines

Yes, you are right.

Quoted 2 lines

Can you try if following patch fix your problem?

This patch can also fix my problem, and indeed is a more secure way.

Thanks for the test, for a better solution, I think 'blkcg_mutex' can be used to protect 'blkg->q_node' list instead of 'queue_lock', so that the 'restart' can be removed because softlockup can be avoided.

Thanks,
Kuai

Quoted 25 lines

Thanks,
Tao

Quoted 15 lines

index 1c1ebeb51003..0ecb4cce8af2 100644

block/blk-cgroup.c +3 -0
@@ -527,6 +527,9 @@ static void blkg_destroy_all(struct gendisk *disk)
         list_for_each_entry_safe(blkg, n, &q->blkg_list, q_node) {
                 struct blkcg *blkcg = blkg->blkcg;
 
+               if (hlist_unhashed(&blkg->blkcg_node))
+                       continue;
+
                 spin_lock(&blkcg->lock);
                 blkg_destroy(blkg);
                 spin_unlock(&blkcg->lock);
Quoted 3 lines

block/blk-cgroup.c static void blkg_destroy_all(struct gendisk *disk) · line 535

Quoted 3 lines
block/blk-cgroup.c +0 -0
   		spin_unlock(&blkcg->lock);
   		/*
 

.

#4 lore raw

Hi,

Quoted 46 lines · Yu Kuai

Hi,

Quoted 43 lines · Tao Su
Quoted 19 lines · Yu Kuai

Hi,

Quoted 16 lines · Tao Su

Kernel hang when poweroff or reboot, due to infinite restart in function blkg_destroy_all. It will goto restart label when a batch of blkgs are destroyed, but not remove blkg node in blkg_list. So the blkg_list is same in every 'restart' and result in kernel hang.

By adding list_del to remove blkg node after destroying, can solve this kernel hang issue and satisfy the previous will to 'restart'.

---
   block/blk-cgroup.c | 1 +

   1 file changed, 1 insertion(+)
Quoted 12 lines

block/blk-cgroup.c static void blkg_destroy_all(struct gendisk *disk) · line 532

Quoted 8 lines
block/blk-cgroup.c +1 -0
index bd50b55bdb61..960eb538a704 100644
@@ -530,6 +530,7 @@ static void blkg_destroy_all(struct gendisk *disk)
            spin_lock(&blkcg->lock);
            blkg_destroy(blkg);
+        list_del(&blkg->q_node);

blkg should stay on the queue list until blkg_free_workfn(), otherwise parent blkg can be freed before child, which will cause some known issue.

Yes, directly removing blkg node is not appropriate, which I noticed some comments in blkg_destroy(), thanks for pointing out this issue.

Quoted 3 lines

I think this hung happens when total blkg is greater than BLKG_DESTROY_BATCH_SIZE, right?

Yes, you are right.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Can you try if following patch fix your problem?

This patch can also fix my problem, and indeed is a more secure way.

Thanks for the test, for a better solution, I think 'blkcg_mutex' can be used to protect 'blkg->q_node' list instead of 'queue_lock', so that the 'restart' can be removed because softlockup can be avoided.

I looked into this, and I found that this is not a easy thing to do.

Anyway, feel free to submit a new patch based on my orignial suggestion.

Thanks,
Kuai

author
#5 lore raw
Quoted 54 lines · Yu Kuai

Hi,

Quoted 51 lines · Yu Kuai

Hi,

Quoted 47 lines · Tao Su
Quoted 20 lines · Yu Kuai

Hi,

Quoted 17 lines · Tao Su

Kernel hang when poweroff or reboot, due to infinite restart in function
blkg_destroy_all. It will goto restart label when a batch of blkgs are destroyed, but not remove blkg node in blkg_list. So the blkg_list is same in every 'restart' and result in kernel hang.

By adding list_del to remove blkg node after destroying, can solve this kernel hang issue and satisfy the previous will to 'restart'.

---
   block/blk-cgroup.c | 1 +

   1 file changed, 1 insertion(+)
Quoted 12 lines

block/blk-cgroup.c static void blkg_destroy_all(struct gendisk *disk) · line 532

Quoted 8 lines
block/blk-cgroup.c +1 -0
index bd50b55bdb61..960eb538a704 100644
@@ -530,6 +530,7 @@ static void blkg_destroy_all(struct gendisk *disk)
            spin_lock(&blkcg->lock);
            blkg_destroy(blkg);
+        list_del(&blkg->q_node);

blkg should stay on the queue list until blkg_free_workfn(), otherwise parent blkg can be freed before child, which will cause some known issue.

Yes, directly removing blkg node is not appropriate, which I noticed some comments in blkg_destroy(), thanks for pointing out this issue.

Quoted 3 lines

I think this hung happens when total blkg is greater than BLKG_DESTROY_BATCH_SIZE, right?

Yes, you are right.

Quoted 2 lines

Can you try if following patch fix your problem?

This patch can also fix my problem, and indeed is a more secure way.

Quoted 8 lines
Quoted 4 lines

Thanks for the test, for a better solution, I think 'blkcg_mutex' can be used to protect 'blkg->q_node' list instead of 'queue_lock', so that the 'restart' can be removed because softlockup can be avoided.

I looked into this, and I found that this is not a easy thing to do.

Anyway, feel free to submit a new patch based on my orignial suggestion.

Thanks for your contribution and careful review, I will submit the new patch if no other comments.

Thanks,
Tao

Quoted 4 lines

Thanks,
Kuai