unmount livelock due to racing with bdi-flusher threads
Folks,
I just had an umount take a very long time burning a CPU the entire time. It wasn't the unmount thread, either, it was the the bdi flusher thread for the the filesystem being unmounted. It was spinning with this perf top trace:
553144.00 76.9% writeback_inodes_wb [kernel.kallsyms]
106434.00 14.8% __ticket_spin_lock [kernel.kallsyms]
25646.00 3.6% __ticket_spin_unlock [kernel.kallsyms]
10512.00 1.5% _raw_spin_lock [kernel.kallsyms]
9606.00 1.3% put_super [kernel.kallsyms]
7920.00 1.1% __put_super [kernel.kallsyms]
5592.00 0.8% down_read_trylock [kernel.kallsyms]
46.00 0.0% kfree [kernel.kallsyms]
22.00 0.0% __do_softirq [kernel.kallsyms]
19.00 0.0% wb_writeback [kernel.kallsyms]
16.00 0.0% wb_do_writeback [kernel.kallsyms]
8.00 0.0% queue_io [kernel.kallsyms]
6.00 0.0% run_timer_softirq [kernel.kallsyms]
6.00 0.0% local_bh_enable_ip [kernel.kallsyms]This went on for ~7m25s (according to the pmchart trace I had on screen) before something broke the livelock by writing the inodes to disk (maybe the xfssyncd) and the unmount then completed a couple of seconds later.
Quoted 1 line
From the above profile, I'm assuming that
writeback_inodes_wb()was
seeing pin_sb_for_writeback(sb) failing and moving dirty inodes from the the b_io to the b_more_io list, then being called again, splicing the inodes on b_more_io back to b_io, and then failed again to pin_sb_for_writeback() for each inode, moving them back to the b_more_io list....
This is on 2.6.36-rc1 + the radix tree fixes for writeback.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
ping?
Quoted 47 lines · Dave Chinner
Folks,
I just had an umount take a very long time burning a CPU the entire time. It wasn't the unmount thread, either, it was the the bdi flusher thread for the the filesystem being unmounted. It was spinning with this perf top trace:
553144.00 76.9% writeback_inodes_wb [kernel.kallsyms] 106434.00 14.8% __ticket_spin_lock [kernel.kallsyms] 25646.00 3.6% __ticket_spin_unlock [kernel.kallsyms] 10512.00 1.5% _raw_spin_lock [kernel.kallsyms] 9606.00 1.3% put_super [kernel.kallsyms] 7920.00 1.1% __put_super [kernel.kallsyms] 5592.00 0.8% down_read_trylock [kernel.kallsyms] 46.00 0.0% kfree [kernel.kallsyms] 22.00 0.0% __do_softirq [kernel.kallsyms] 19.00 0.0% wb_writeback [kernel.kallsyms] 16.00 0.0% wb_do_writeback [kernel.kallsyms] 8.00 0.0% queue_io [kernel.kallsyms] 6.00 0.0% run_timer_softirq [kernel.kallsyms] 6.00 0.0% local_bh_enable_ip [kernel.kallsyms]This went on for ~7m25s (according to the pmchart trace I had on screen) before something broke the livelock by writing the inodes to disk (maybe the xfssyncd) and the unmount then completed a couple of seconds later.
From the above profile, I'm assuming that
writeback_inodes_wb()was seeingpin_sb_for_writeback(sb)failing and moving dirty inodes from the theb_ioto theb_more_iolist, then being called again, splicing the inodes onb_more_ioback tob_io, and then failed again topin_sb_for_writeback()for each inode, moving them back to theb_more_iolist....This is on 2.6.36-rc1 + the radix tree fixes for writeback.
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Dave Chinner [email protected]
Jan Kara
reviewerQuoted 1 line · Dave Chinner
ping?
Pong ;) I finally had a look at this. Thanks for reporting this.
Quoted 25 lines
Quoted 25 lines
I just had an umount take a very long time burning a CPU the entire time. It wasn't the unmount thread, either, it was the the bdi flusher thread for the the filesystem being unmounted. It was spinning with this perf top trace:
553144.00 76.9% writeback_inodes_wb [kernel.kallsyms] 106434.00 14.8% __ticket_spin_lock [kernel.kallsyms] 25646.00 3.6% __ticket_spin_unlock [kernel.kallsyms] 10512.00 1.5% _raw_spin_lock [kernel.kallsyms] 9606.00 1.3% put_super [kernel.kallsyms] 7920.00 1.1% __put_super [kernel.kallsyms] 5592.00 0.8% down_read_trylock [kernel.kallsyms] 46.00 0.0% kfree [kernel.kallsyms] 22.00 0.0% __do_softirq [kernel.kallsyms] 19.00 0.0% wb_writeback [kernel.kallsyms] 16.00 0.0% wb_do_writeback [kernel.kallsyms] 8.00 0.0% queue_io [kernel.kallsyms] 6.00 0.0% run_timer_softirq [kernel.kallsyms] 6.00 0.0% local_bh_enable_ip [kernel.kallsyms]This went on for ~7m25s (according to the pmchart trace I had on screen) before something broke the livelock by writing the inodes to disk (maybe the xfssyncd) and the unmount then completed a couple of seconds later.
Quoted 8 lines
Quoted 8 lines
From the above profile, I'm assuming that
writeback_inodes_wb()was seeingpin_sb_for_writeback(sb)failing and moving dirty inodes from the theb_ioto theb_more_iolist, then being called again, splicing the inodes onb_more_ioback tob_io, and then failed again topin_sb_for_writeback()for each inode, moving them back to theb_more_iolist....This is on 2.6.36-rc1 + the radix tree fixes for writeback.
Indeed, your analysis looks correct. The trouble is following:
Flusher thread Umount
- start processing background writeback
- get s_mount for writing
- queue syncing work for flusher
- waits until flusher thread gets to it
- loops infinitely, trying to get s_umount
for reading
In principle a classical ABBA deadlock. Actually, there are more complicated (and harder to hit) cases like:
Flusher thread Sync Remount
- processes background
writeback
- gets s_umount for reading
- queues syncing work
- waits for syncing work
- tries to get s_umount for writing and blocks
- now loops infinitely
since it cannot get
s_umount for reading anymore
The question is how to properly resolve it. The cases like the second one above show that it's not enough to just somehow work-around writeback during umount. Also it's not only background writeback that can get deadlocked like this but generally anything submitted via __bdi_start_writeback (as these kinds of writeback do not have superblock specified).
I think the best resolution of this problem would be to change the work that is submitted via bdi_start_writeback() (i.e., the work without superblock = work which needs to do locking) to "target based scheme" like Christoph wanted already some time ago. I actually have a patch to do this for background writeback so I will just modify it to apply to a wider range of writeback as well. Or Christoph, do you already have some patches in this direction?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 7 lines · Jan Kara
I think the best resolution of this problem would be to change the work that is submitted via
bdi_start_writeback()(i.e., the work without superblock = work which needs to do locking) to "target based scheme" like Christoph wanted already some time ago. I actually have a patch to do this for background writeback so I will just modify it to apply to a wider range of writeback as well. Or Christoph, do you already have some patches in this direction?
I've not started any work on it yet.