Skip to contentSkip to conversation

unmount livelock due to racing with bdi-flusher threads

2.6.36-rc1 Dave Chinner 3 participants 4 messages Aug 21, 2010 – Oct 1, 2010 VFS

lore
Share & tools
lore
#0 lore raw

Folks,

I just had an umount take a very long time burning a CPU the entire time. It wasn't the unmount thread, either, it was the the bdi flusher thread for the the filesystem being unmounted. It was spinning with this perf top trace:

           553144.00 76.9% writeback_inodes_wb  [kernel.kallsyms]
           106434.00 14.8% __ticket_spin_lock   [kernel.kallsyms]
            25646.00  3.6% __ticket_spin_unlock [kernel.kallsyms]
            10512.00  1.5% _raw_spin_lock       [kernel.kallsyms]
             9606.00  1.3% put_super            [kernel.kallsyms]
             7920.00  1.1% __put_super          [kernel.kallsyms]
             5592.00  0.8% down_read_trylock    [kernel.kallsyms]
               46.00  0.0% kfree                [kernel.kallsyms]
               22.00  0.0% __do_softirq         [kernel.kallsyms]
               19.00  0.0% wb_writeback         [kernel.kallsyms]
               16.00  0.0% wb_do_writeback      [kernel.kallsyms]
                8.00  0.0% queue_io             [kernel.kallsyms]
                6.00  0.0% run_timer_softirq    [kernel.kallsyms]
                6.00  0.0% local_bh_enable_ip   [kernel.kallsyms]

This went on for ~7m25s (according to the pmchart trace I had on screen) before something broke the livelock by writing the inodes to disk (maybe the xfssyncd) and the unmount then completed a couple of seconds later.

Quoted 1 line

From the above profile, I'm assuming that writeback_inodes_wb() was

seeing pin_sb_for_writeback(sb) failing and moving dirty inodes from the the b_io to the b_more_io list, then being called again, splicing the inodes on b_more_io back to b_io, and then failed again to pin_sb_for_writeback() for each inode, moving them back to the b_more_io list....

This is on 2.6.36-rc1 + the radix tree fixes for writeback.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#1 lore raw

ping?

Quoted 47 lines · Dave Chinner

Folks,

I just had an umount take a very long time burning a CPU the entire time. It wasn't the unmount thread, either, it was the the bdi flusher thread for the the filesystem being unmounted. It was spinning with this perf top trace:

           553144.00 76.9% writeback_inodes_wb  [kernel.kallsyms]
           106434.00 14.8% __ticket_spin_lock   [kernel.kallsyms]
            25646.00  3.6% __ticket_spin_unlock [kernel.kallsyms]
            10512.00  1.5% _raw_spin_lock       [kernel.kallsyms]
             9606.00  1.3% put_super            [kernel.kallsyms]
             7920.00  1.1% __put_super          [kernel.kallsyms]
             5592.00  0.8% down_read_trylock    [kernel.kallsyms]
               46.00  0.0% kfree                [kernel.kallsyms]
               22.00  0.0% __do_softirq         [kernel.kallsyms]
               19.00  0.0% wb_writeback         [kernel.kallsyms]
               16.00  0.0% wb_do_writeback      [kernel.kallsyms]
                8.00  0.0% queue_io             [kernel.kallsyms]
                6.00  0.0% run_timer_softirq    [kernel.kallsyms]
                6.00  0.0% local_bh_enable_ip   [kernel.kallsyms]

This went on for ~7m25s (according to the pmchart trace I had on screen) before something broke the livelock by writing the inodes to disk (maybe the xfssyncd) and the unmount then completed a couple of seconds later.

From the above profile, I'm assuming that writeback_inodes_wb() was seeing pin_sb_for_writeback(sb) failing and moving dirty inodes from the the b_io to the b_more_io list, then being called again, splicing the inodes on b_more_io back to b_io, and then failed again to pin_sb_for_writeback() for each inode, moving them back to the b_more_io list....

This is on 2.6.36-rc1 + the radix tree fixes for writeback.

Cheers,

Dave.

Signature · 6 lines
Dave Chinner
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Dave Chinner
[email protected]
reviewer
#2 lore raw
Quoted 1 line · Dave Chinner

ping?

Pong ;) I finally had a look at this. Thanks for reporting this.

Quoted 25 lines
Quoted 25 lines

I just had an umount take a very long time burning a CPU the entire time. It wasn't the unmount thread, either, it was the the bdi flusher thread for the the filesystem being unmounted. It was spinning with this perf top trace:

           553144.00 76.9% writeback_inodes_wb  [kernel.kallsyms]
           106434.00 14.8% __ticket_spin_lock   [kernel.kallsyms]
            25646.00  3.6% __ticket_spin_unlock [kernel.kallsyms]
            10512.00  1.5% _raw_spin_lock       [kernel.kallsyms]
             9606.00  1.3% put_super            [kernel.kallsyms]
             7920.00  1.1% __put_super          [kernel.kallsyms]
             5592.00  0.8% down_read_trylock    [kernel.kallsyms]
               46.00  0.0% kfree                [kernel.kallsyms]
               22.00  0.0% __do_softirq         [kernel.kallsyms]
               19.00  0.0% wb_writeback         [kernel.kallsyms]
               16.00  0.0% wb_do_writeback      [kernel.kallsyms]
                8.00  0.0% queue_io             [kernel.kallsyms]
                6.00  0.0% run_timer_softirq    [kernel.kallsyms]
                6.00  0.0% local_bh_enable_ip   [kernel.kallsyms]

This went on for ~7m25s (according to the pmchart trace I had on screen) before something broke the livelock by writing the inodes to disk (maybe the xfssyncd) and the unmount then completed a couple of seconds later.

Quoted 8 lines
Quoted 8 lines

From the above profile, I'm assuming that writeback_inodes_wb() was seeing pin_sb_for_writeback(sb) failing and moving dirty inodes from the the b_io to the b_more_io list, then being called again, splicing the inodes on b_more_io back to b_io, and then failed again to pin_sb_for_writeback() for each inode, moving them back to the b_more_io list....

This is on 2.6.36-rc1 + the radix tree fixes for writeback.

Indeed, your analysis looks correct. The trouble is following:

Flusher thread Umount
- start processing background writeback
- get s_mount for writing
- queue syncing work for flusher
- waits until flusher thread gets to it
- loops infinitely, trying to get s_umount
for reading

In principle a classical ABBA deadlock. Actually, there are more complicated (and harder to hit) cases like:

Flusher thread Sync Remount
- processes background
writeback
- gets s_umount for reading
- queues syncing work
- waits for syncing work
- tries to get s_umount for writing and blocks
- now loops infinitely
since it cannot get
s_umount for reading anymore

The question is how to properly resolve it. The cases like the second one above show that it's not enough to just somehow work-around writeback during umount. Also it's not only background writeback that can get deadlocked like this but generally anything submitted via __bdi_start_writeback (as these kinds of writeback do not have superblock specified).

I think the best resolution of this problem would be to change the work that is submitted via bdi_start_writeback() (i.e., the work without superblock = work which needs to do locking) to "target based scheme" like Christoph wanted already some time ago. I actually have a patch to do this for background writeback so I will just modify it to apply to a wider range of writeback as well. Or Christoph, do you already have some patches in this direction?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#3 lore raw
Quoted 7 lines · Jan Kara

I think the best resolution of this problem would be to change the work that is submitted via bdi_start_writeback() (i.e., the work without superblock = work which needs to do locking) to "target based scheme" like Christoph wanted already some time ago. I actually have a patch to do this for background writeback so I will just modify it to apply to a wider range of writeback as well. Or Christoph, do you already have some patches in this direction?

I've not started any work on it yet.