bcache: avoid unnecessary soft lockup in kworker update_writeback_rate()
PATCH 1/1 bcache: avoid unnecessary soft lockup in kworker update_writeback_rate()
The kworker routine update_writeback_rate() is schedued to update the writeback rate in every 5 seconds by default. Before calling __update_writeback_rate() to do real job, semaphore dc->writeback_lock should be held by the kworker routine.
At the same time, bcache writeback thread routine bch_writeback_thread() also needs to hold dc->writeback_lock before flushing dirty data back into the backing device. If the dirty data set is large, it might be very long time for bch_writeback_thread() to scan all dirty buckets and releases dc->writeback_lock. In such case update_writeback_rate() can be starved for long enough time so that kernel reports a soft lockup warn- ing started like:
watchdog: BUG: soft lockup - CPU#246 stuck for 23s! [kworker/246:31:179713]
Such soft lockup condition is unnecessary, because after the writeback thread finishes its job and releases dc->writeback_lock, the kworker update_writeback_rate() may continue to work and everything is fine indeed.
This patch avoids the unnecessary soft lockup by the following method,
- Add new members to struct cached_dev
- dc->retry_nr (0 by default)
- dc->retry_max (6 by default)
- In update_writeback_rate() call down_read_trylock(&dc->writeback_lock) firstly, if it fails then lock contention happens. If dc->retry_nr is smaller than dc->retry_max, increase 1 to dc->retry_nr, and reschedule the kworker to retry after a bit long time.
- If lock contention happens and dc->retry_nr is equal to dc->retry_max, no retry anymore and call down_read(&dc->writeback_lock) to wait for the lock.
By the above method, at worst case update_writeback_rate() may retry for 2+ minutes before blocking on dc->writeback_lock by calling down_read(). For a 4TB cache device with 1TB dirty data, 90%+ of the unnecessary soft lockup warning message can be avoided.
When retrying to acquire dc->writeback_lock in update_writeback_rate(), of course the writeback rate cannot be updated. It is fair, because when the kworker is blocked on the lock contention of dc->writeback_lock, the writeback rate cannot be updated neither.
--- drivers/md/bcache/bcache.h | 7 +++++ drivers/md/bcache/writeback.c | 49 +++++++++++++++++++++++++++++++---- 2 files changed, 51 insertions(+), 5 deletions(-)
drivers/md/bcache/bcache.h +7 -0
@@ -395,6 +395,13 @@ struct cached_dev {atomic_t io_errors;unsigned int error_limit;unsigned int offline_seconds; + +/* +* Retry to update writeback_rate if contention happens for +* down_read(dc->writeback_lock) in update_writeback_rate() +*/ +unsigned int retry_nr; +unsigned int retry_max; };enum alloc_reserve {
drivers/md/bcache/writeback.c +44 -5
83 lines not on this page · raw message
Signature · 1 line
2.34.1
Quoted 11 lines
The kworker routine
update_writeback_rate()is schedued to update the writeback rate in every 5 seconds by default. Before calling__update_writeback_rate()to do real job, semaphoredc->writeback_lockshould be held by the kworker routine.At the same time, bcache writeback thread routine
bch_writeback_thread()also needs to holddc->writeback_lockbefore flushing dirty data back into the backing device. If the dirty data set is large, it might be very long time forbch_writeback_thread()to scan all dirty buckets and releasesdc->writeback_lock.
Hi Coly,cached_dev_write() needs dc->writeback_lock, if the writeback thread
holds writeback_lock too long, high write IO latency may happen. I wonder
if it is a nicer way to limit the scale of the scanning in writeback. For example,
just scan 512GB in stead of the whole cache disk。
Quoted 5 lines · 李磊
Quoted 5 lines
The kworker routine
update_writeback_rate()is schedued to update the writeback rate in every 5 seconds by default. Before calling__update_writeback_rate()to do real job, semaphoredc->writeback_lockshould be held by the kworker routine.
Quoted 8 lines
Quoted 5 lines
At the same time, bcache writeback thread routine
bch_writeback_thread()also needs to holddc->writeback_lockbefore flushing dirty data back into the backing device. If the dirty data set is large, it might be very long time forbch_writeback_thread()to scan all dirty buckets and releasesdc->writeback_lock.Hi Coly,
cached_dev_write()needsdc->writeback_lock, if the writeback thread
holdswriteback_locktoo long, high write IO latency may happen. I wonder
From my observation, such situation happens in one of the last scan before all dirty data gets flushed. If the cache device is very large, and dirty keys are only a few, such scan will take quite long time.
It wasn't a problem years ago, but currently it is easy to have a 10TB+ cache device, now the latency is observed.
Quoted 3 lines
if it is a nicer way to limit the scale of the scanning in writeback. For example,
just scan 512GB in stead of the whole cache disk。
Scan each 512GB space doesn't help too much. Because current btree iteration code doesn't handle continue-from-where-stopped very well, next time when continue form where stoppped, the previous key might be invalided already.
An ideal way, might be split the single large btree into multiple ones. People suggested me to divide the single tree into e.g. 64 or 128 trees, and only lock a single tree when doing writeback on gc on one of the trees. Maybe now it is about time to think over it again...
Coly Li
Quoted 33 lines · 李磊
Quoted 5 lines
Quoted 5 lines
The kworker routine
update_writeback_rate()is schedued to update the writeback rate in every 5 seconds by default. Before calling__update_writeback_rate()to do real job, semaphoredc->writeback_lockshould be held by the kworker routine.Quoted 8 lines
Quoted 5 lines
At the same time, bcache writeback thread routine
bch_writeback_thread()also needs to holddc->writeback_lockbefore flushing dirty data back into the backing device. If the dirty data set is large, it might be very long time forbch_writeback_thread()to scan all dirty buckets and releasesdc->writeback_lock.Hi Coly,
cached_dev_write()needsdc->writeback_lock, if the writeback thread
holdswriteback_locktoo long, high write IO latency may happen. I wonderFrom my observation, such situation happens in one of the last scan
before all dirty data gets flushed. If the cache device is very large, and dirty keys are only a few, such scan will take quite long time.It wasn't a problem years ago, but currently it is easy to have a 10TB+ cache device, now the latency is observed.
Quoted 3 lines
if it is a nicer way to limit the scale of the scanning in writeback. For example,
just scan 512GB in stead of the whole cache disk。Scan each 512GB space doesn't help too much. Because current btree iteration code doesn't handle continue-from-where-stopped very well, next time when continue form where stoppped, the previous key might be invalided already.
Quoted 5 lines
An ideal way, might be split the single large btree into multiple ones. People suggested me to divide the single tree into e.g. 64 or 128 trees, and only lock a single tree when doing writeback on gc on one of the trees. Maybe now it is about time to think over it again...
I got it. Thanks for the detailed explanation.