filesystems, btrfs, cgroups, debugging tools
Hi everyone,
I'm really looking forward to LSF/MM this year. I can bring along a fair amount of data from production about benchmarking and stability.
We've been expanding our btrfs rollout, and we're also fixing up priority inversions when cgroup IO controllers are put in place. I think we have btrfs fixed up, but ext4 seems to be incompatible with IO controllers due to data=ordered IO. We haven't tried XFS with the controllers yet but I don't think there will be any major blockers there.
I'm also hoping filesystem slab shrinking gets into the agenda, since we have a few ugly hacks there to keep production happy.
Thanks,
Chris
Jan Kara
reviewerHi Chris,
Quoted 7 lines · Chris Mason
I'm really looking forward to LSF/MM this year. I can bring along a fair amount of data from production about benchmarking and stability.
We've been expanding our btrfs rollout, and we're also fixing up priority inversions when cgroup IO controllers are put in place. I think we have btrfs fixed up, but ext4 seems to be incompatible with IO controllers due to data=ordered IO.
Yeah, I suspect I know what you hit but still I'd be interested in hearing more details about your usecase and the problems you see. Maybe it could be helped.
Quoted 5 lines
We haven't tried XFS with the controllers yet but I don't think there will be any major blockers there.
I'm also hoping filesystem slab shrinking gets into the agenda, since we have a few ugly hacks there to keep production happy.
I'm interested in this as well but I suspect we need someone willing to spend a long time with slab reclaim guts to improve this. And so far nobody was bothered enough...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Hi Jan,
Quoted 6 lines · Jan Kara
Hi Chris,
Quoted 3 lines · Chris Mason
I'm really looking forward to LSF/MM this year. I can bring along a fair amount of data from production about benchmarking and stability.
Quoted 9 lines
Quoted 4 lines
We've been expanding our btrfs rollout, and we're also fixing up priority inversions when cgroup IO controllers are put in place. I think we have btrfs fixed up, but ext4 seems to be incompatible with IO controllers due to data=ordered IO.
Yeah, I suspect I know what you hit but still I'd be interested in hearing more details about your usecase and the problems you see. Maybe it could be helped.
Both btrfs and ext4 are root drive filesystems for us. The IO controller is basically making sure the root drive isn't saturated by lower priority tasks, which might be anything from system updates to log files to actually part of the workload.
With ext4, the data=ordered IO done during transaction commits makes priority inversions that I don't see a way around. It's dramatically better than ext3, but still happens enough that we can't enforce IO limits at all. It really only takes one low prio IO to sneak into kjournald's list to wreck everything.
Quoted 9 lines
Quoted 5 lines
We haven't tried XFS with the controllers yet but I don't think there will be any major blockers there.
I'm also hoping filesystem slab shrinking gets into the agenda, since we have a few ugly hacks there to keep production happy.
I'm interested in this as well but I suspect we need someone willing to spend a long time with slab reclaim guts to improve this. And so far nobody was bothered enough...
Yeah, we'll keep chipping away at this one, but it's going to take a while.
-chris
Jan Kara
reviewerHi Chris,
Quoted 22 lines · Chris Mason
Quoted 6 lines · Jan Kara
Hi Chris,
Quoted 3 lines · Chris Mason
I'm really looking forward to LSF/MM this year. I can bring along a fair amount of data from production about benchmarking and stability.
Quoted 9 lines
Quoted 4 lines
We've been expanding our btrfs rollout, and we're also fixing up priority inversions when cgroup IO controllers are put in place. I think we have btrfs fixed up, but ext4 seems to be incompatible with IO controllers due to data=ordered IO.
Yeah, I suspect I know what you hit but still I'd be interested in hearing more details about your usecase and the problems you see. Maybe it could be helped.
Both btrfs and ext4 are root drive filesystems for us. The IO controller is basically making sure the root drive isn't saturated by lower priority tasks, which might be anything from system updates to log files to actually part of the workload.
Quoted 5 lines
With ext4, the data=ordered IO done during transaction commits makes priority inversions that I don't see a way around. It's dramatically better than ext3, but still happens enough that we can't enforce IO limits at all. It really only takes one low prio IO to sneak into kjournald's list to wreck everything.
AFAIU we could do a similar thing like what Tejun implemented for btrfs metadata where the submitter can override blkcg to which the IO is accounted. In ext4's case if kjournald is doing the writeback, it would get accounted to the root blkcg. It will allow containers to somewhat violate the bounds set to their blkcg but the priority inversion should be rarer - sadly we cannot easily make it completely go away as if the original process not only attaches the inode to the transaction but also submits the data blocks with low priority, transaction commit still has to wait for this IO to complete so the whole commit will be still blocked.
So probably a better fix would be to introduce another data journalling mode for ext4 where we'd unconditionally use unwritten extents for data writeback. We actually have it implemented in ext4 hidden behind 'dioread_nolock' mount option but it needs more polishing and possibly testing.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 10 lines · Jan Kara
Hi Chris,
Quoted 6 lines · Chris Mason
With ext4, the data=ordered IO done during transaction commits makes priority inversions that I don't see a way around. It's dramatically better than ext3, but still happens enough that we can't enforce IO limits at all. It really only takes one low prio IO to sneak into kjournald's list to wreck everything.
Quoted 9 lines
AFAIU we could do a similar thing like what Tejun implemented for btrfs metadata where the submitter can override blkcg to which the IO is accounted. In ext4's case if kjournald is doing the writeback, it would get accounted to the root blkcg. It will allow containers to somewhat violate the bounds set to their blkcg but the priority inversion should be rarer - sadly we cannot easily make it completely go away as if the original process not only attaches the inode to the transaction but also submits the data blocks with low priority, transaction commit still has to wait for this IO to complete so the whole commit will be still blocked.
Yeah, I think this was the problem we hit. balance_dirty_pages and friends will trigger low priority write back, and if kjournald ends up waiting on that, we're out of luck.
Quoted 6 lines
So probably a better fix would be to introduce another data journalling mode for ext4 where we'd unconditionally use unwritten extents for data writeback. We actually have it implemented in ext4 hidden behind '
dioread_nolock' mount option but it needs more polishing and possibly testing.
I wonder how that compares in performance to my old data=guarded idea. I think a better step one might be to add tracepoints when blocks are added to the ordered list, so we can better understand if we're adding them in error. It felt like it was happening more often than it should.
On the FB side, we found one more prio inversion in btrfs from the free space cache (IO going down as data instead of metadata) and we're testing the fix for that. It should hopefully be the last one, and then we can compare how effective the different options are.
-chris
Jan Kara
reviewerQuoted 38 lines · Chris Mason
Quoted 11 lines · Jan Kara
Quoted 9 lines · Chris Mason
With ext4, the data=ordered IO done during transaction commits makes priority inversions that I don't see a way around. It's dramatically better
than ext3, but still happens enough that we can't enforce IO limits at all.
It really only takes one low prio IO to sneak into kjournald's list to wreck
everything.Quoted 14 lines
AFAIU we could do a similar thing like what Tejun implemented for btrfs metadata where the submitter can override blkcg to which the IO is accounted. In ext4's case if kjournald is doing the writeback, it would get
accounted to the root blkcg. It will allow containers to somewhat violate
the bounds set to their blkcg but the priority inversion should be rarer -
sadly we cannot easily make it completely go away as if the original process
not only attaches the inode to the transaction but also submits the data blocks with low priority, transaction commit still has to wait for this IO to
complete so the whole commit will be still blocked.Yeah, I think this was the problem we hit.
balance_dirty_pagesand friends will trigger low priority write back, and if kjournald ends up waiting on that, we're out of luck.Quoted 6 lines
So probably a better fix would be to introduce another data journalling mode for ext4 where we'd unconditionally use unwritten extents for data writeback. We actually have it implemented in ext4 hidden behind '
dioread_nolock' mount option but it needs more polishing and possibly testing.
Quoted 4 lines
I wonder how that compares in performance to my old data=guarded idea. I think a better step one might be to add tracepoints when blocks are added to the ordered list, so we can better understand if we're adding them in error. It felt like it was happening more often than it should.
In ext4 / jbd2 this mechanism is actually different from ext3. We don't track individual blocks in ordered list anymore, we just track inodes and flush all mapped & dirty pages from those inodes when doing transaction commit. It does potentially imply more work but the locking imposed by the original jbd scheme was incompatible with reasonably efficient writeback and generally very special data writeback path causing various troubles.
We do take care to add inode to a transaction only when we allocate new block for it so in this sence we shouldn't ever be adding them "in error". You are right that we could optimize this by doing a special handling for writeback of blocks beyond end of file (which is what data=guarded was about if I remember right) however with delayed allocation you then have to be careful not to flush new inode size to disk before flushing the data blocks so it gets more complicated.
But creating unwritten extents and converting them to written on IO completion certainly has non-negligible cost (that's why this is not the default yet) so possibly the complication is worth it.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 59 lines · Jan Kara
Quoted 38 lines · Chris Mason
Quoted 11 lines · Jan Kara
Quoted 9 lines · Chris Mason
With ext4, the data=ordered IO done during transaction commits makes priority inversions that I don't see a way around. It's dramatically better
than ext3, but still happens enough that we can't enforce IO limits at all.
It really only takes one low prio IO to sneak into kjournald's list to wreck
everything.Quoted 14 lines
AFAIU we could do a similar thing like what Tejun implemented for btrfs metadata where the submitter can override blkcg to which the IO is accounted. In ext4's case if kjournald is doing the writeback, it would get
accounted to the root blkcg. It will allow containers to somewhat violate
the bounds set to their blkcg but the priority inversion should be rarer -
sadly we cannot easily make it completely go away as if the original process
not only attaches the inode to the transaction but also submits the data blocks with low priority, transaction commit still has to wait for this IO to
complete so the whole commit will be still blocked.Yeah, I think this was the problem we hit.
balance_dirty_pagesand friends will trigger low priority write back, and if kjournald ends up waiting on that, we're out of luck.Quoted 6 lines
So probably a better fix would be to introduce another data journalling mode for ext4 where we'd unconditionally use unwritten extents for data writeback. We actually have it implemented in ext4 hidden behind '
dioread_nolock' mount option but it needs more polishing and possibly testing.Quoted 4 lines
I wonder how that compares in performance to my old data=guarded idea. I think a better step one might be to add tracepoints when blocks are added to the ordered list, so we can better understand if we're adding them in error. It felt like it was happening more often than it should.
In ext4 / jbd2 this mechanism is actually different from ext3. We don't track individual blocks in ordered list anymore, we just track inodes and flush all mapped & dirty pages from those inodes when doing transaction commit. It does potentially imply more work but the locking imposed by the original jbd scheme was incompatible with reasonably efficient writeback and generally very special data writeback path causing various troubles.
We do take care to add inode to a transaction only when we allocate new block for it so in this sence we shouldn't ever be adding them "in error". You are right that we could optimize this by doing a special handling for writeback of blocks beyond end of file (which is what data=guarded was about if I remember right) however with delayed allocation you then have to be careful not to flush new inode size to disk before flushing the data blocks so it gets more complicated.
Quoted 4 lines
But creating unwritten extents and converting them to written on IO completion certainly has non-negligible cost (that's why this is not the default yet) so possibly the complication is worth it.
This isn't too far from what btrfs does, where we just update the metadata to point to the new blocks after the IO is done. data=guarded was similar but much more of a hack ;)
-chris
Quoted 7 lines · Chris Mason
Hi everyone,
I'm really looking forward to LSF/MM this year. I can bring along a fair amount of data from production about benchmarking and stability.
We've been expanding our btrfs rollout, and we're also fixing up priority inversions when cgroup IO controllers are put in place. I think we have btrfs fixed up, but ext4 seems to be incompatible with IO controllers due to data=ordered IO. We haven't tried XFS with the controllers yet but I don't think there will be any major blockers there.
I'm also hoping filesystem slab shrinking gets into the agenda, since we have a few ugly hacks there to keep production happy.
Hi,
What about the "debugging tools" part of $Subject?
thanks,
Signature · 1 line
~Randy
Quoted 12 lines · Randy Dunlap
Quoted 7 lines · Chris Mason
Hi everyone,
I'm really looking forward to LSF/MM this year. I can bring along a fair amount of data from production about benchmarking and stability.
We've been expanding our btrfs rollout, and we're also fixing up priority inversions when cgroup IO controllers are put in place. I think we have btrfs fixed up, but ext4 seems to be incompatible with IO controllers due to data=ordered IO. We haven't tried XFS with the controllers yet but I don't think there will be any major blockers there.
I'm also hoping filesystem slab shrinking gets into the agenda, since we have a few ugly hacks there to keep production happy.
Hi,
What about the "debugging tools" part of $Subject?
We're mostly adapting and extending bpf as we find fun ways to use it for tracing, error injection etc. I'm more a consumer of other people's work here, but I'm happy to join in discussions in this area and talk about the tools we're using most often.
Today I debugged an early ENOSPC problem without even adding a printk. The future feels weird.
-chris
Quoted 11 lines · Chris Mason
Quoted 9 lines · Randy Dunlap
Quoted 7 lines · Chris Mason
Hi everyone,
I'm really looking forward to LSF/MM this year. I can bring along a fair amount of data from production about benchmarking and stability.
We've been expanding our btrfs rollout, and we're also fixing up priority inversions when cgroup IO controllers are put in place. I think we have btrfs fixed up, but ext4 seems to be incompatible with IO controllers due to data=ordered IO. We haven't tried XFS with the controllers yet but I don't think there will be any major blockers there.
I'm also hoping filesystem slab shrinking gets into the agenda, since we have a few ugly hacks there to keep production happy.
Quoted 7 lines
Quoted 3 lines
Hi,
What about the "debugging tools" part of $Subject?We're mostly adapting and extending bpf as we find fun ways to use it for tracing, error injection etc. I'm more a consumer of other people's work here, but I'm happy to join in discussions in this area and talk about the tools we're using most often.
Today I debugged an early ENOSPC problem without even adding a printk. The future feels weird.
That would be interesting (not that I'll be there).
thanks,
Signature · 1 line
~Randy