Skip to contentSkip to conversation

blk-cgroup: Optimize blkcg_rstat_flush()

PATCH In review Waiman Long 2 participants 4 messages May 31, 2022

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 blk-cgroup: Optimize blkcg_rstat_flush()

For a system with many CPUs and block devices, the time to do blkcg_rstat_flush() from cgroup_rstat_flush() can be rather long. It can be especially problematic as interrupt is disabled during the flush. It was reported that it might take seconds in some extreme cases leading to hard lockup messages.

As it is likely that not all the percpu blkg_iostat_set's has been updated since the last flush, those stale blkg_iostat_set's don't need to be flushed in this case. This patch optimizes blkcg_rstat_flush() by checking the current sequence number against the one recorded since the last flush and skip the blkg_iostat_set if the sequence number hasn't changed. There is a slight chance that it may miss an update that is being done in parallel, the new update will just have to wait until the next flush.

---
 block/blk-cgroup.c | 18 +++++++++++++++---
 block/blk-cgroup.h |  1 +
 2 files changed, 16 insertions(+), 3 deletions(-)
block/blk-cgroup.c +15 -3
index 40161a3f68d0..79b89af61ef2 100644
@@ -864,11 +864,23 @@ static void blkcg_rstat_flush(struct cgroup_subsys_state *css, int cpu)
 		unsigned long flags;
 		unsigned int seq;
 
+		seq = u64_stats_fetch_begin(&bisc->sync);
+		/*
+		 * If the sequence number hasn't been updated since the last
+		 * flush, we can skip this blkg_iostat_set though we may miss
+		 * an update that is happening in parallel.
+		 */
+		if (seq == bisc->last_seq)
+			continue;
+
 		/* fetch the current per-cpu values */
-		do {
-			seq = u64_stats_fetch_begin(&bisc->sync);
+		while (true) {
+			bisc->last_seq = seq;
 			blkg_iostat_set(&cur, &bisc->cur);
-		} while (u64_stats_fetch_retry(&bisc->sync, seq));
+			if (!u64_stats_fetch_retry(&bisc->sync, seq))
+				break;
+			seq = u64_stats_fetch_begin(&bisc->sync);
+		}
 
 		/* propagate percpu delta to global */
 		flags = u64_stats_update_begin_irqsave(&blkg->iostat.sync);
block/blk-cgroup.h +1 -0
index d4de0a35e066..22b4ea139b93 100644
@@ -45,6 +45,7 @@ struct blkg_iostat_set {
 	struct u64_stats_sync		sync;
 	struct blkg_iostat		cur;
 	struct blkg_iostat		last;
+	unsigned int			last_seq;
 };
 
 /* association between a blk cgroup and a request queue */
Signature · 1 line
2.31.1
maintainer
#1 lore raw

Hello, Waiman.

Quoted 27 lines · Waiman Long

For a system with many CPUs and block devices, the time to do blkcg_rstat_flush() from cgroup_rstat_flush() can be rather long. It can be especially problematic as interrupt is disabled during the flush. It was reported that it might take seconds in some extreme cases leading to hard lockup messages.

As it is likely that not all the percpu blkg_iostat_set's has been updated since the last flush, those stale blkg_iostat_set's don't need to be flushed in this case. This patch optimizes blkcg_rstat_flush() by checking the current sequence number against the one recorded since the last flush and skip the blkg_iostat_set if the sequence number hasn't changed. There is a slight chance that it may miss an update that is being done in parallel, the new update will just have to wait until the next flush.

---
 block/blk-cgroup.c | 18 +++++++++++++++---
 block/blk-cgroup.h |  1 +
 2 files changed, 16 insertions(+), 3 deletions(-)
block/blk-cgroup.c +0 -0
index 40161a3f68d0..79b89af61ef2 100644
@@ -864,11 +864,23 @@ static void blkcg_rstat_flush(struct cgroup_subsys_state *css, int cpu)
 		unsigned long flags;

block/blk-cgroup.c static void blkcg_rstat_flush(struct cgroup_subsys_state *css, int cpu) · line 874

Quoted 10 lines
block/blk-cgroup.c +8 -0
 		unsigned int seq;
 
+		seq = u64_stats_fetch_begin(&bisc->sync);
+		/*
+		 * If the sequence number hasn't been updated since the last
+		 * flush, we can skip this blkg_iostat_set though we may miss
+		 * an update that is happening in parallel.
+		 */
+		if (seq == bisc->last_seq)
+			continue;

Is this a sufficient solution? The code assumes that there aren't too many blkgs for the cgroup, which can be wrong in some cases. Wouldn't it be better to create a list of updated blkg's per blkcg so that we don't walk all the dormant ones?

Thanks.

Signature · 1 line
tejun
#2 lore raw
Quoted 32 lines · Tejun Heo

Hello, Waiman.

Quoted 29 lines · Waiman Long

For a system with many CPUs and block devices, the time to do blkcg_rstat_flush() from cgroup_rstat_flush() can be rather long. It can be especially problematic as interrupt is disabled during the flush. It was reported that it might take seconds in some extreme cases leading to hard lockup messages.

As it is likely that not all the percpu blkg_iostat_set's has been updated since the last flush, those stale blkg_iostat_set's don't need to be flushed in this case. This patch optimizes blkcg_rstat_flush() by checking the current sequence number against the one recorded since the last flush and skip the blkg_iostat_set if the sequence number hasn't changed. There is a slight chance that it may miss an update that is being done in parallel, the new update will just have to wait until the next flush.

---
  block/blk-cgroup.c | 18 +++++++++++++++---
  block/blk-cgroup.h |  1 +
  2 files changed, 16 insertions(+), 3 deletions(-)
block/blk-cgroup.c +0 -0
index 40161a3f68d0..79b89af61ef2 100644
@@ -864,11 +864,23 @@ static void blkcg_rstat_flush(struct cgroup_subsys_state *css, int cpu)
  		unsigned long flags;
  		unsigned int seq;
 
Quoted 12 lines

block/blk-cgroup.c static void blkcg_rstat_flush(struct cgroup_subsys_state *css, int cpu) · line 874

Quoted 8 lines
block/blk-cgroup.c +8 -0
+		seq = u64_stats_fetch_begin(&bisc->sync);
+		/*
+		 * If the sequence number hasn't been updated since the last
+		 * flush, we can skip this blkg_iostat_set though we may miss
+		 * an update that is happening in parallel.
+		 */
+		if (seq == bisc->last_seq)
+			continue;

Is this a sufficient solution? The code assumes that there aren't too many blkgs for the cgroup, which can be wrong in some cases. Wouldn't it be better to create a list of updated blkg's per blkcg so that we don't walk all the dormant ones?

It is probably not a sufficient solution, but it is simple. The problem with keeping a list of recently updated blkg's is that sequence lock does not provide enough synchronization on the read side to guarantee a race free reset of the list. It may be doable, but I need to think harder on the best way to do it without too much overhead.

Thanks,
Longman

#3 lore raw
Quoted 46 lines · Waiman Long
Quoted 36 lines · Tejun Heo

Hello, Waiman.

Quoted 33 lines · Waiman Long

For a system with many CPUs and block devices, the time to do blkcg_rstat_flush() from cgroup_rstat_flush() can be rather long. It can be especially problematic as interrupt is disabled during the flush. It was reported that it might take seconds in some extreme cases leading to hard lockup messages.

As it is likely that not all the percpu blkg_iostat_set's has been updated since the last flush, those stale blkg_iostat_set's don't need to be flushed in this case. This patch optimizes blkcg_rstat_flush() by checking the current sequence number against the one recorded since the last flush and skip the blkg_iostat_set if the sequence number hasn't changed. There is a slight chance that it may miss an update that is being done in parallel, the new update will just have to wait until the next flush.

---
  block/blk-cgroup.c | 18 +++++++++++++++---
  block/blk-cgroup.h |  1 +

  2 files changed, 16 insertions(+), 3 deletions(-)
block/blk-cgroup.c +3 -0
index 40161a3f68d0..79b89af61ef2 100644
@@ -864,11 +864,23 @@ static void blkcg_rstat_flush(struct 
 cgroup_subsys_state *css, int cpu)
           unsigned long flags;
           unsigned int seq;
   +        seq = u64_stats_fetch_begin(&bisc->sync);
+        /*
+         * If the sequence number hasn't been updated since the last
+         * flush, we can skip this blkg_iostat_set though we may miss
Quoted 8 lines

block/blk-cgroup.c static void blkcg_rstat_flush(struct cgroup_subsys_state *css, int cpu) · line 874

Quoted 4 lines
block/blk-cgroup.c +4 -0
+         * an update that is happening in parallel.
+         */
+        if (seq == bisc->last_seq)
+            continue;

Is this a sufficient solution? The code assumes that there aren't too many blkgs for the cgroup, which can be wrong in some cases. Wouldn't it be better to create a list of updated blkg's per blkcg so that we don't walk all the dormant ones?

Quoted 5 lines

It is probably not a sufficient solution, but it is simple. The problem with keeping a list of recently updated blkg's is that sequence lock does not provide enough synchronization on the read side to guarantee a race free reset of the list. It may be doable, but I need to think harder on the best way to do it without too much overhead.

Thanks,
Longman

I think we can use a lockless list to keep a list of recently updated blkg for each blkcg. There is an atomic operation overhead when an entry is added to the list though. I will send an updated patch later today or tomorrow.

Thanks,
Longman