Skip to contentSkip to conversation

block: Revert "let blkcg_gq grab request queue's refcnt"

PATCH In review Bart Van Assche 2 participants 7 messages Jan 30, 2023 – Feb 1, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 block: Revert "let blkcg_gq grab request queue's refcnt"

Since commit 0a9a25ca7843 ("block: let blkcg_gq grab request queue's refcnt") for many request queues the reference count drops to 1 when the request queue is destroyed instead of to 0. In other words, the request queue is leaked. Fix this by reverting that commit.

This leak was discovered by running the following shell command before and after the sub-page block layer tests:

cat /sys/kernel/debug/block/sub_page_limit_queues

Without this patch, the above debugfs attribute is increased by one after each such sub-page block layer test. With this revert applied, that debugfs attribute drops to zero after each sub-page block layer test.

Cc Ming Lei <[email protected]>
Cc Christoph Hellwig <[email protected]>
---
 block/blk-cgroup.c | 4 ----
 1 file changed, 4 deletions(-)
block/blk-cgroup.c +0 -4
index cb110fc51940..2e531268f725 100644
@@ -141,7 +141,6 @@ static void blkg_free_workfn(struct work_struct *work)
 	if (q) {
 		list_del_init(&blkg->q_node);
 		mutex_unlock(&q->blkcg_mutex);
-		blk_put_queue(q);
 	}
 
 	free_percpu(blkg->iostat_cpu);
@@ -273,9 +272,6 @@ static struct blkcg_gq *blkg_alloc(struct blkcg *blkcg, struct gendisk *disk,
 	if (!blkg->iostat_cpu)
 		goto err_free;
 
-	if (!blk_get_queue(disk->queue))
-		goto err_free;
-
 	blkg->q = disk->queue;
 	INIT_LIST_HEAD(&blkg->q_node);
 	spin_lock_init(&blkg->async_bio_lock);
#1 lore raw

Hi Bart,

Quoted 4 lines · Bart Van Assche

Since commit 0a9a25ca7843 ("block: let blkcg_gq grab request queue's refcnt") for many request queues the reference count drops to 1 when the request queue is destroyed instead of to 0. In other words, the request queue is leaked. Fix this by reverting that commit.

When/where you observe that the reference count drops to 1 instead of 0?

Do you have kmem leak log?

Probably, the last drop is in blkg_free_workfn().

Thanks,
Ming

#2 lore raw
Quoted 8 lines · Ming Lei

Hi Bart,

Quoted 4 lines · Bart Van Assche

Since commit 0a9a25ca7843 ("block: let blkcg_gq grab request queue's refcnt") for many request queues the reference count drops to 1 when the request queue is destroyed instead of to 0. In other words, the request queue is leaked. Fix this by reverting that commit.

Quoted 5 lines

When/where you observe that the reference count drops to 1 instead of 0?

Do you have kmem leak log?

Probably, the last drop is in blkg_free_workfn().

Hi Ming,

The reference count leak was discovered while I was testing my patch series that adds support for sub-page limits (https://lore.kernel.org/linux-block/[email protected]/T/#t). The second patch in that series adds a counter that tracks the number of queues that need support for limits below the page size (sub_page_limit_queues). I noticed that without this patch that counter increases but never decreases. With this patch applied, that counter drops back to zero after having run a test that needs support for sub-page limits.

Thanks,

Bart.

#3 lore raw
Quoted 17 lines · Bart Van Assche
Quoted 8 lines · Ming Lei

Hi Bart,

Quoted 4 lines · Bart Van Assche

Since commit 0a9a25ca7843 ("block: let blkcg_gq grab request queue's refcnt") for many request queues the reference count drops to 1 when the request queue is destroyed instead of to 0. In other words, the request queue is leaked. Fix this by reverting that commit.

Quoted 5 lines

When/where you observe that the reference count drops to 1 instead of 0?

Do you have kmem leak log?

Probably, the last drop is in blkg_free_workfn().

Hi Ming,

Quoted 7 lines

The reference count leak was discovered while I was testing my patch series
that adds support for sub-page limits (https://lore.kernel.org/linux-block/[email protected]/T/#t).
The second patch in that series adds a counter that tracks the number of
queues that need support for limits below the page size
(sub_page_limit_queues). I noticed that without this patch that counter
increases but never decreases. With this patch applied, that counter drops
back to zero after having run a test that needs support for sub-page limits.

I can reproduce the issue by scsi_debug now, but blkg_release() isn't called, so looks like one blkcg_gq lifetime issue since blkcg_exit_disk() is really run.

Thanks,
Ming

#4 lore raw
Quoted 26 lines · Ming Lei
Quoted 17 lines · Bart Van Assche
Quoted 8 lines · Ming Lei

Hi Bart,

Quoted 4 lines · Bart Van Assche

Since commit 0a9a25ca7843 ("block: let blkcg_gq grab request queue's refcnt") for many request queues the reference count drops to 1 when the request queue is destroyed instead of to 0. In other words, the request queue is leaked. Fix this by reverting that commit.

Quoted 5 lines

When/where you observe that the reference count drops to 1 instead of 0?

Do you have kmem leak log?

Probably, the last drop is in blkg_free_workfn().

Hi Ming,

Quoted 7 lines

The reference count leak was discovered while I was testing my patch series
that adds support for sub-page limits (https://lore.kernel.org/linux-block/[email protected]/T/#t).
The second patch in that series adds a counter that tracks the number of
queues that need support for limits below the page size
(sub_page_limit_queues). I noticed that without this patch that counter
increases but never decreases. With this patch applied, that counter drops
back to zero after having run a test that needs support for sub-page limits.

Quoted 3 lines

I can reproduce the issue by scsi_debug now, but blkg_release() isn't called, so looks like one blkcg_gq lifetime issue since blkcg_exit_disk() is really run.

The problem is caused by 3b8cc6298724 ("blk-cgroup: Optimize blkcg_rstat_flush()").

This commit will hold blkg instance until blkcg_rstat_flush() is called, and which may be delayed to css_release_work_fn().

Thanks,
Ming

#5 lore raw
Quoted 27 lines · Ming Lei
Quoted 26 lines · Ming Lei
Quoted 24 lines · Bart Van Assche
Quoted 8 lines · Ming Lei

Hi Bart,

Quoted 4 lines · Bart Van Assche

Since commit 0a9a25ca7843 ("block: let blkcg_gq grab request queue's refcnt") for many request queues the reference count drops to 1 when the request queue is destroyed instead of to 0. In other words, the request queue is leaked. Fix this by reverting that commit.

Quoted 5 lines

When/where you observe that the reference count drops to 1 instead of 0?

Do you have kmem leak log?

Probably, the last drop is in blkg_free_workfn().

Hi Ming,

The reference count leak was discovered while I was testing my patch series
that adds support for sub-page limits (https://lore.kernel.org/linux-block/[email protected]/T/#t).
The second patch in that series adds a counter that tracks the number of
queues that need support for limits below the page size
(sub_page_limit_queues). I noticed that without this patch that counter
increases but never decreases. With this patch applied, that counter drops
back to zero after having run a test that needs support for sub-page limits.

Quoted 8 lines
Quoted 3 lines

I can reproduce the issue by scsi_debug now, but blkg_release() isn't called, so looks like one blkcg_gq lifetime issue since blkcg_exit_disk() is really run.

The problem is caused by 3b8cc6298724 ("blk-cgroup: Optimize blkcg_rstat_flush()").

This commit will hold blkg instance until blkcg_rstat_flush() is called, and which may be delayed to css_release_work_fn().

The following patch can address the blkg leak issue:

block/blk-cgroup.c +4 -0
index cb110fc51940..78f855c34746 100644
@@ -2034,6 +2034,10 @@ void blk_cgroup_bio_start(struct bio *bio)
 	struct blkg_iostat_set *bis;
 	unsigned long flags;
 
+	/* Root-level stats are sourced from system-wide IO stats */
+	if (!cgroup_parent(blkcg->css.cgroup))
+		return;
+
 	cpu = get_cpu();
 	bis = per_cpu_ptr(bio->bi_blkg->iostat_cpu, cpu);
 	flags = u64_stats_update_begin_irqsave(&bis->sync);

Thanks,
Ming

author Tested-by
#6 lore raw
Quoted 7 lines · Ming Lei

The following patch can address the blkg leak issue:

block/blk-cgroup.c +0 -0
index cb110fc51940..78f855c34746 100644

block/blk-cgroup.c void blk_cgroup_bio_start(struct bio *bio) · line 2043

Quoted 10 lines
block/blk-cgroup.c +4 -0
@@ -2034,6 +2034,10 @@ void blk_cgroup_bio_start(struct bio *bio)
  	struct blkg_iostat_set *bis;
  	unsigned long flags;
 
+	/* Root-level stats are sourced from system-wide IO stats */
+	if (!cgroup_parent(blkcg->css.cgroup))
+		return;
+
  	cpu = get_cpu();
  	bis = per_cpu_ptr(bio->bi_blkg->iostat_cpu, cpu);
  	flags = u64_stats_update_begin_irqsave(&bis->sync);

The above patch passes my tests. Feel free to add: