Updated performance results
I have re-run the raid tests with re-creating the fileset between each of the random write workloads and performance does now match the previous newformat results. The bad news is that the huge gain that I had attributed to the newformat release, does not really exist. All of the previous results(except for the newformat run) were not re-creating the fileset, so the gain in performance was due only to having a fresh set of files, not any code changes.
So, I have done 2 new sets of runs to look into this further. One is a 3 hour run of single threaded random write to the RAID system. I have compared this to ext3. Performance results are here: http://btrfs.boxacle.net/repository/raid/longwrite/longwrite/Longrandomwrite.html
and graphing of all the iostat data can be found here:
http://btrfs.boxacle.net/repository/raid/longwrite/summary.html
The iostat graphs for btrfs are interesting for a number of reasons. First, it takes about 3000 seconds (or 50 minutes) for btrfs to reach steady state. Second, if you look at write throughput from the device view vs. the btrfs/application view, we see that for a application throughput of 21.5MB/sec it requires 63MB/sec of actual disk writes. That is an overhead of 3 to 1 vs an overhead of ~0 for ext3. Also, looking at the change in iops vs MB/sec, we see that while btrfs starts out with reasonable size IOs, it quickly deteriorate to an average IO size of only 13kb. Remember, the starting file set is only 100GB on a 2.1TB filesystem, and all data is overwrite, and this is single threaded, so there is no reason this should fragment. It seems like the allocator is having a problem doing sequential allocations.
Another set of runs I did was to do repetitive 5 minute random write runs. Results are here:
http://btrfs.boxacle.net/repository/raid/repeat/repeat/repeat.html
This shows the dramatic degredation after just a short time, but I believe there is a fair amount of overhead in btrfs after newly mounting the FS (which this test did between each run) so I repeated without unmounting and remounting and results are here:
http://btrfs.boxacle.net/repository/raid/repeat-nomount/repeat/repeat-nomount.html
These results show a much less dramatic degradation, but btrfs still degrades by over 40% in just 30 minutes of run time. In fact, it was still degrading by 10% every 5 minutes when this test ended.
Steve
Quoted 7 lines · Steven Pratt
I have re-run the raid tests with re-creating the fileset between each of the random write workloads and performance does now match the previous newformat results. The bad news is that the huge gain that I had attributed to the newformat release, does not really exist. All of the previous results(except for the newformat run) were not re-creating the fileset, so the gain in performance was due only to having a fresh set of files, not any code changes.
Thanks for doing all of these runs. This is still a little different than what I have here, my initial runs are very very fast and after 10 or so level out to a relatively low performance on random writes. With nodatacow, it stays even.
Quoted 10 lines
So, I have done 2 new sets of runs to look into this further. One is a 3 hour run of single threaded random write to the RAID system. I have compared this to ext3. Performance results are here:
http://btrfs.boxacle.net/repository/raid/longwrite/longwrite/Longrandomwrite.htmland graphing of all the iostat data can be found here:
http://btrfs.boxacle.net/repository/raid/longwrite/summary.html
Quoted 12 lines
The iostat graphs for btrfs are interesting for a number of reasons. First, it takes about 3000 seconds (or 50 minutes) for btrfs to reach steady state. Second, if you look at write throughput from the device view vs. the btrfs/application view, we see that for a application throughput of 21.5MB/sec it requires 63MB/sec of actual disk writes. That is an overhead of 3 to 1 vs an overhead of ~0 for ext3. Also, looking at the change in iops vs MB/sec, we see that while btrfs starts out with reasonable size IOs, it quickly deteriorate to an average IO size of only 13kb. Remember, the starting file set is only 100GB on a 2.1TB filesystem, and all data is overwrite, and this is single threaded, so there is no reason this should fragment. It seems like the allocator is having a problem doing sequential allocations.
There are two things happening. First the default allocation scheme isn't very well suited to this, mount -o ssd will perform better. But over the long term, random overwrites to the file cause a lot of writes to the extent allocation tree. That's really what -o nodatacow is saving us. There are optimizations we can do, but we're holding off on that in favor of enospc and other pressing things.
But, with all of that said, Josef has some really important allocator improvements. I've put them out along with our pending patches into the experimental branch of the btrfs-unstable tree. Could you please give this branch a try both with and without the ssd mount option?
-chris
Quoted 11 lines · Chris Mason
Quoted 2 lines · Steven Pratt
I have re-run the raid tests with re-creating the fileset between each of the random write workloads and performance does now match the previous newformat results. The bad news is that the huge gain that I had attributed to the newformat release, does not really exist. All of the previous results(except for the newformat run) were not re-creating the fileset, so the gain in performance was due only to having a fresh set of files, not any code changes.
Thanks for doing all of these runs. This is still a little different than what I have here, my initial runs are very very fast and after 10 or so level out to a relatively low performance on random writes. With nodatacow, it stays even.
Right, I do not see this problem with nodatacow.
Quoted 9 lines
Quoted 8 lines
So, I have done 2 new sets of runs to look into this further. One is a 3 hour run of single threaded random write to the RAID system. I have compared this to ext3. Performance results are here: http://btrfs.boxacle.net/repository/raid/longwrite/longwrite/Longrandomwrite.html
and graphing of all the iostat data can be found here:
http://btrfs.boxacle.net/repository/raid/longwrite/summary.html
The iostat graphs for btrfs are interesting for a number of reasons. First, it takes about 3000 seconds (or 50 minutes) for btrfs to reach steady state. Second, if you look at write throughput from the device view vs. the btrfs/application view, we see that for a application throughput of 21.5MB/sec it requires 63MB/sec of actual disk writes. That is an overhead of 3 to 1 vs an overhead of ~0 for ext3. Also, looking at the change in iops vs MB/sec, we see that while btrfs starts out with reasonable size IOs, it quickly deteriorate to an average IO size of only 13kb. Remember, the starting file set is only 100GB on a 2.1TB filesystem, and all data is overwrite, and this is single threaded, so there is no reason this should fragment. It seems like the allocator is having a problem doing sequential allocations.
Quoted 7 lines
There are two things happening. First the default allocation scheme isn't very well suited to this, mount -o ssd will perform better. But over the long term, random overwrites to the file cause a lot of writes to the extent allocation tree. That's really what -o nodatacow is saving us. There are optimizations we can do, but we're holding off on that in favor of enospc and other pressing things.
Well I have -o ssd data that I can upload, but it was worse than without. I do understand about timing and priorities.
Quoted 6 lines
But, with all of that said, Josef has some really important allocator improvements. I've put them out along with our pending patches into the experimental branch of the btrfs-unstable tree. Could you please give this branch a try both with and without the ssd mount option?
Sure, will try to get to it tomorrow.
Steve
Quoted 3 lines
-chris
Quoted 54 lines · Steven Pratt
Quoted 11 lines · Chris Mason
Quoted 8 lines · Steven Pratt
I have re-run the raid tests with re-creating the fileset between each of the random write workloads and performance does now match the previous newformat results. The bad news is that the huge gain that I had attributed to the newformat release, does not really exist. All of the previous results(except for the newformat run) were not re-creating the fileset, so the gain in performance was due only to having a fresh set of files, not any code changes.
Quoted 6 lines
Thanks for doing all of these runs. This is still a little different than what I have here, my initial runs are very very fast and after 10 or so level out to a relatively low performance on random writes. With nodatacow, it stays even.
Right, I do not see this problem with nodatacow.
Quoted 24 lines
Quoted 9 lines
So, I have done 2 new sets of runs to look into this further. One is a 3 hour run of single threaded random write to the RAID system. I have compared this to ext3. Performance results are here:
http://btrfs.boxacle.net/repository/raid/longwrite/longwrite/Longrandomwrite.htmland graphing of all the iostat data can be found here:
http://btrfs.boxacle.net/repository/raid/longwrite/summary.html
Quoted 14 lines
The iostat graphs for btrfs are interesting for a number of reasons.
First, it takes about 3000 seconds (or 50 minutes) for btrfs to
reach steady state. Second, if you look at write throughput from the device view vs. the btrfs/application view, we see that for a application throughput of 21.5MB/sec it requires 63MB/sec of actual
disk writes. That is an overhead of 3 to 1 vs an overhead of ~0 for
ext3. Also, looking at the change in iops vs MB/sec, we see that while btrfs starts out with reasonable size IOs, it quickly deteriorate to an average IO size of only 13kb. Remember, the starting file set is only 100GB on a 2.1TB filesystem, and all data is overwrite, and this is single threaded, so there is no reason this should fragment. It seems like the allocator is having a problem doing sequential allocations.Quoted 7 lines
There are two things happening. First the default allocation scheme isn't very well suited to this, mount -o ssd will perform better. But over the long term, random overwrites to the file cause a lot of writes to the extent allocation tree. That's really what -o nodatacow is saving us. There are optimizations we can do, but we're holding off on that in favor of enospc and other pressing things.
Well I have -o ssd data that I can upload, but it was worse than without. I do understand about timing and priorities.
Quoted 7 lines
Quoted 6 lines
But, with all of that said, Josef has some really important allocator improvements. I've put them out along with our pending patches into the experimental branch of the btrfs-unstable tree. Could you please give this branch a try both with and without the ssd mount option?
Sure, will try to get to it tomorrow.
Sorry, I missed a fix in the experimental branch. I'll push out a rebased version in a few minutes.
-chris
Quoted 6 lines · Chris Mason
Quoted 2 lines
Quoted 1 line
Sure, will try to get to it tomorrow.
Sorry, I missed a fix in the experimental branch. I'll push out a rebased version in a few minutes.
Ok, the rebased version is ready to use.
-chris
Quoted 4 lines · Chris Mason
Quoted 2 lines · Chris Mason
Quoted 2 lines
Quoted 2 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Sure, will try to get to it tomorrow.
Sorry, I missed a fix in the experimental branch. I'll push out a rebased version in a few minutes.
Ok, the rebased version is ready to use.
Ok, good. Also seems I misspoke on the -o ssd results. I looked them over again this morning and they are slightly bette than without. Initial score of 46MB/sec vs 44MB/sec without, and degrades after 30 minutes to about 25MB/sec vs 20MB/sec without. So after 30 minutes of runtime -o ssd is running about 25% faster, but still significant degrade.
Steve
Quoted 7 lines
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · Chris Mason
Quoted 2 lines · Chris Mason
Quoted 2 lines
Quoted 2 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Sure, will try to get to it tomorrow.
Sorry, I missed a fix in the experimental branch. I'll push out a rebased version in a few minutes.
Ok, the rebased version is ready to use.
New results are up for both with and without nodatacow. Not much change.
http://btrfs.boxacle.net/repository/raid/history/History.html
Have another run going with nodatacow and ssd.
Steve
Quoted 7 lines
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Steven Pratt
Quoted 6 lines · Chris Mason
Quoted 4 lines · Chris Mason
Quoted 1 line
Sure, will try to get to it tomorrow.
Sorry, I missed a fix in the experimental branch. I'll push out a rebased version in a few minutes.
Quoted 6 lines
Quoted 1 line
Ok, the rebased version is ready to use.
New results are up for both with and without nodatacow. Not much change.
http://btrfs.boxacle.net/repository/raid/history/History.html
Have another run going with nodatacow and ssd.
Hi Steve,
I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.
The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?
-chris
Quoted 18 lines · Chris Mason
Quoted 16 lines · Steven Pratt
Quoted 10 lines · Chris Mason
Quoted 6 lines · Chris Mason
Quoted 2 lines
Sure, will try to get to it tomorrow.
Sorry, I missed a fix in the experimental branch. I'll push out a rebased version in a few minutes.
Ok, the rebased version is ready to use.
New results are up for both with and without nodatacow. Not much change.
http://btrfs.boxacle.net/repository/raid/history/History.html
Quoted 9 lines
Quoted 2 lines
Have another run going with nodatacow and ssd.
Hi Steve,
I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.
Sure, no problem.
Quoted 4 lines
The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?
Yes, thanks for pointing that out, had so many issues I forgot to update the graphs for other file systems. Just pushed new graphs with data for 2.6.30-rc7 for all the other file systems. This was from your "newformat" branch from June 6th.
Steve
Quoted 2 lines
-chris
Quoted 8 lines · Steven Pratt
Quoted 6 lines
Hi Steve,
I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.
Sure, no problem.
Quoted 7 lines
Quoted 3 lines
The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?
Yes, thanks for pointing that out, had so many issues I forgot to update the graphs for other file systems. Just pushed new graphs with data for 2.6.30-rc7 for all the other file systems. This was from your "newformat" branch from June 6th.
I've been tuning the 128 thread large file streaming writes, and found some easy optimizations. While I'm fixing up these patches, could you please do a streaming O_DIRECT write test run for me? I think buffered writeback in general has some problems right now on high end arrays.
On my box 2.6.31-rc5 streaming buffered write with xfs only got at 200MB/s (with the 128 thread ffsb workload). Buffered btrfs goes at 175MB/s.
O_DIRECT btrfs runs at 390MB/s, while XFS varies a bit between 330MB/s and 250MB/s.
I'm using a 1MB write blocksize.
-chris
HI,
Do you have any benchmarks against non-raid common workloads? Like say a desktop user? It would be great to compare against ext3, ext4, xfs etc.,
Thanks,
Quoted 38 lines · Chris Mason
Quoted 8 lines · Steven Pratt
Quoted 6 lines
Hi Steve,
I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.
Sure, no problem.
Quoted 7 lines
Quoted 3 lines
The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?
Yes, thanks for pointing that out, had so many issues I forgot to update the graphs for other file systems. Just pushed new graphs with data for 2.6.30-rc7 for all the other file systems. This was from your "newformat" branch from June 6th.
I've been tuning the 128 thread large file streaming writes, and found some easy optimizations. While I'm fixing up these patches, could you please do a streaming
O_DIRECTwrite test run for me? I think buffered writeback in general has some problems right now on high end arrays.On my box 2.6.31-rc5 streaming buffered write with xfs only got at 200MB/s (with the 128 thread ffsb workload). Buffered btrfs goes at 175MB/s.
O_DIRECTbtrfs runs at 390MB/s, while XFS varies a bit between 330MB/s and 250MB/s.I'm using a 1MB write blocksize.
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · debian developer
HI,
Do you have any benchmarks against non-raid common workloads? Like say a desktop user? It would be great to compare against ext3, ext4, xfs etc.,
Yes, have had a little trouble with that box recently, but plenty of results based on the 2.6.29 kernels here:
http://btrfs.boxacle.net/repository/single-disk/History/History.html
If you are not familiar with the runs I have been doing, you can find the details of the benchmarking machine and test procedures here: http://btrfs.boxacle.net/
Steve
Quoted 46 lines
Thanks,
Quoted 42 lines · Chris Mason
Quoted 9 lines · Steven Pratt
Quoted 6 lines
Hi Steve,
I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.
Sure, no problem.
Quoted 9 lines
Quoted 4 lines
The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?
Yes, thanks for pointing that out, had so many issues I forgot to update the graphs for other file systems. Just pushed new graphs with data for 2.6.30-rc7 for all the other file systems. This was from your "newformat" branch from June 6th.
I've been tuning the 128 thread large file streaming writes, and found some easy optimizations. While I'm fixing up these patches, could you please do a streaming
O_DIRECTwrite test run for me? I think buffered writeback in general has some problems right now on high end arrays.On my box 2.6.31-rc5 streaming buffered write with xfs only got at 200MB/s (with the 128 thread ffsb workload). Buffered btrfs goes at 175MB/s.
O_DIRECTbtrfs runs at 390MB/s, while XFS varies a bit between 330MB/s and 250MB/s.I'm using a 1MB write blocksize.
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 26 lines · Chris Mason
Quoted 9 lines · Steven Pratt
Quoted 6 lines
Hi Steve,
I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.
Sure, no problem.
Quoted 9 lines
Quoted 4 lines
The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?
Yes, thanks for pointing that out, had so many issues I forgot to update the graphs for other file systems. Just pushed new graphs with data for 2.6.30-rc7 for all the other file systems. This was from your "newformat" branch from June 6th.
I've been tuning the 128 thread large file streaming writes, and found some easy optimizations. While I'm fixing up these patches, could you please do a streaming
O_DIRECTwrite test run for me? I think buffered writeback in general has some problems right now on high end arrays.
Quoted 9 lines
On my box 2.6.31-rc5 streaming buffered write with xfs only got at 200MB/s (with the 128 thread ffsb workload). Buffered btrfs goes at 175MB/s.
O_DIRECTbtrfs runs at 390MB/s, while XFS varies a bit between 330MB/s and 250MB/s.I'm using a 1MB write blocksize.
On my todo list, but am swamped this week trying to get ready for vacation. Will try to get to it as soon as I can.
Stee
Quoted 2 lines
-chris
Quoted 26 lines · Steven Pratt
Quoted 25 lines · Chris Mason
Quoted 7 lines · Steven Pratt
Quoted 5 lines
Hi Steve,
I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.
Sure, no problem.
Quoted 7 lines
Quoted 3 lines
The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?
Yes, thanks for pointing that out, had so many issues I forgot to update the graphs for other file systems. Just pushed new graphs with data for 2.6.30-rc7 for all the other file systems. This was from your "newformat" branch from June 6th.
I've been tuning the 128 thread large file streaming writes, and found some easy optimizations. While I'm fixing up these patches, could you please do a streaming
O_DIRECTwrite test run for me? I think buffered writeback in general has some problems right now on high end arrays.On my box 2.6.31-rc5 streaming buffered write with xfs only got at 200MB/s (with the 128 thread ffsb workload). Buffered btrfs goes at 175MB/s.
Quoted 6 lines
Quoted 4 lines
O_DIRECTbtrfs runs at 390MB/s, while XFS varies a bit between 330MB/s and 250MB/s.I'm using a 1MB write blocksize.
On my todo list, but am swamped this week trying to get ready for vacation. Will try to get to it as soon as I can.
Ok, I've pushed out a very raw version of my buffered write fixes to a new branch named performance on btrfs-unstable.
Please try this with the streaming large file create workload. I'm also curious to see if it improves on your box when you mount with
mount -o thread_pool=128
-chris
Quoted 43 lines · Chris Mason
Quoted 32 lines · Steven Pratt
Quoted 30 lines · Chris Mason
Quoted 13 lines · Steven Pratt
Quoted 6 lines
Hi Steve,
I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.
Sure, no problem.
Quoted 4 lines
The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?
Quoted 5 lines
Yes, thanks for pointing that out, had so many issues I forgot to update the graphs for other file systems. Just pushed new graphs with data for 2.6.30-rc7 for all the other file systems. This was
from your "newformat" branch from June 6th.
I've been tuning the 128 thread large file streaming writes, and found some easy optimizations. While I'm fixing up these patches, could you please do a streaming
O_DIRECTwrite test run for me? I think buffered writeback in general has some problems right now on high end arrays.On my box 2.6.31-rc5 streaming buffered write with xfs only got at 200MB/s (with the 128 thread ffsb workload). Buffered btrfs goes at 175MB/s.
Quoted 8 lines
Quoted 5 lines
O_DIRECTbtrfs runs at 390MB/s, while XFS varies a bit between 330MB/s and 250MB/s.I'm using a 1MB write blocksize.
On my todo list, but am swamped this week trying to get ready for vacation. Will try to get to it as soon as I can.
Quoted 9 lines
Ok, I've pushed out a very raw version of my buffered write fixes to a new branch named performance on btrfs-unstable.
Please try this with the streaming large file create workload. I'm also curious to see if it improves on your box when you mount with
mount -o thread_pool=128
Better late than never. Finally got this finished up. Mixed bag on this one. BTRFS lags significantly on single threaded. Seems unable to keep IO outstanding to the device. Less that 60% busy on the DM device, compared to 97%+ for all other filesystems. nodatacow helps out, increasing utilization to about 70%, but still trails by a large margin.
Results are more favorable for multithreaded tests. nodatacow is actually the top performer here! However, cow still raises it's ugly head and causes significant performance degradation (45%) and increased CPU (43%). Also, even without cow, BTRFS is consuming 8-10x more CPU than other File Systems. I don't have oprofile data for these runs, as that was causing some issues with BTRFS. Will retry, and see if that problem is fixed.
thread_pool seemed to make no difference at all.
All runs were done against an August 20th pull of experimental tree. These are 1M, odirect file creates, with each file being 1G in size.
Results can be found here:
http://btrfs.boxacle.net/repository/raid/large_create_test/write-test/1M_odirect_create.html
Steve
Quoted 2 lines
-chris
Quoted 6 lines · Steven Pratt
Better late than never. Finally got this finished up. Mixed bag on this one. BTRFS lags significantly on single threaded. Seems unable to keep IO outstanding to the device. Less that 60% busy on the DM device, compared to 97%+ for all other filesystems. nodatacow helps out, increasing utilization to about 70%, but still trails by a large margin.
Hi Steve,
Jens Axboe did some profiling on his big test rig and I think we found the biggest CPU problems. The end result is now setting in the master branch of the btrfs-unstable repo.
On his boxes, btrfs went from around 400MB/s streaming writes to 1GB/s limit, and we're now tied with XFS while using less CPU time.
Hopefully you will see similar results ;)
-chris
Quoted 12 lines · Chris Mason
Quoted 7 lines · Steven Pratt
Better late than never. Finally got this finished up. Mixed bag on this one. BTRFS lags significantly on single threaded. Seems unable to keep IO outstanding to the device. Less that 60% busy on the DM device, compared to 97%+ for all other filesystems. nodatacow helps out, increasing utilization to about 70%, but still trails by a large margin.
Hi Steve,
Quoted 9 lines
Jens Axboe did some profiling on his big test rig and I think we found the biggest CPU problems. The end result is now setting in the master branch of the btrfs-unstable repo.
On his boxes, btrfs went from around 400MB/s streaming writes to 1GB/s limit, and we're now tied with XFS while using less CPU time.
Hopefully you will see similar results ;)
Hmmm, well no I didn't. Throughputs at 1 and 128 threads are pretty much unchanged, although I do see a good CPU savings on the 128 thread case (with cow). For 16 threads we actually regressed with cow enabled.
Results are here:
http://btrfs.boxacle.net/repository/raid/large_create_test/write-test/1M_odirect_create.html
I'll try to look more into this next week.
Steve
Quoted 6 lines
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 24 lines · Steven Pratt
Quoted 10 lines · Chris Mason
Quoted 6 lines · Steven Pratt
Better late than never. Finally got this finished up. Mixed bag on this one. BTRFS lags significantly on single threaded. Seems unable to keep IO outstanding to the device. Less that 60% busy on the DM device, compared to 97%+ for all other filesystems. nodatacow helps out, increasing utilization to about 70%, but still trails by a large margin.
Hi Steve,
Quoted 8 lines
Jens Axboe did some profiling on his big test rig and I think we found the biggest CPU problems. The end result is now setting in the master branch of the btrfs-unstable repo.
On his boxes, btrfs went from around 400MB/s streaming writes to 1GB/s limit, and we're now tied with XFS while using less CPU time.
Hopefully you will see similar results ;)
Hmmm, well no I didn't. Throughputs at 1 and 128 threads are pretty much unchanged, although I do see a good CPU savings on the 128 thread case (with cow). For 16 threads we actually regressed with cow enabled.
Quoted 6 lines
Results are here:
http://btrfs.boxacle.net/repository/raid/large_create_test/write-test/1M_odirect_create.html
I'll try to look more into this next week.
Hmmm, Jens was benchmarking buffered writes, but he was also testing on his new per-bdi write back code. If your next run could be buffered instead of O_DIRECT, I'd be curious to see the results.
Thanks,
Chris
Quoted 27 lines · Chris Mason
Quoted 26 lines · Steven Pratt
Quoted 10 lines · Chris Mason
Quoted 6 lines · Steven Pratt
Better late than never. Finally got this finished up. Mixed bag on this one. BTRFS lags significantly on single threaded. Seems unable to keep IO outstanding to the device. Less that 60% busy on the DM device, compared to 97%+ for all other filesystems. nodatacow helps out, increasing utilization to about 70%, but still trails by a large margin.
Hi Steve,
Quoted 8 lines
Jens Axboe did some profiling on his big test rig and I think we found the biggest CPU problems. The end result is now setting in the master branch of the btrfs-unstable repo.
On his boxes, btrfs went from around 400MB/s streaming writes to 1GB/s limit, and we're now tied with XFS while using less CPU time.
Hopefully you will see similar results ;)
Hmmm, well no I didn't. Throughputs at 1 and 128 threads are pretty much unchanged, although I do see a good CPU savings on the 128 thread case (with cow). For 16 threads we actually regressed with cow enabled.
Results are here:
Quoted 8 lines
Quoted 4 lines
http://btrfs.boxacle.net/repository/raid/large_create_test/write-test/1M_odirect_create.html
I'll try to look more into this next week.
Hmmm, Jens was benchmarking buffered writes, but he was also testing on his new per-bdi write back code. If your next run could be buffered instead of
O_DIRECT, I'd be curious to see the results.
I found out today that a larger MAX_WRITEBACK_PAGES is still an essential for me. It basically doubles throughput on btrfs. So I think we need to do something about that, and sooner and rather than later.
Signature · 1 line
Jens Axboe
Quoted 33 lines · Chris Mason
Quoted 31 lines · Steven Pratt
Quoted 11 lines · Chris Mason
Quoted 7 lines · Steven Pratt
Better late than never. Finally got this finished up. Mixed bag on this one. BTRFS lags significantly on single threaded. Seems unable to keep IO outstanding to the device. Less that 60% busy on the DM device, compared to 97%+ for all other filesystems. nodatacow helps out, increasing utilization to about 70%, but still trails by a large margin.
Hi Steve,
Quoted 9 lines
Jens Axboe did some profiling on his big test rig and I think we found the biggest CPU problems. The end result is now setting in the master branch of the btrfs-unstable repo.
On his boxes, btrfs went from around 400MB/s streaming writes to 1GB/s limit, and we're now tied with XFS while using less CPU time.
Hopefully you will see similar results ;)
Hmmm, well no I didn't. Throughputs at 1 and 128 threads are pretty much unchanged, although I do see a good CPU savings on the 128 thread case (with cow). For 16 threads we actually regressed with cow enabled.
Results are here:
http://btrfs.boxacle.net/repository/raid/large_create_test/write-test/1M_odirect_create.html
Quoted 9 lines
Quoted 3 lines
I'll try to look more into this next week.
Hmmm, Jens was benchmarking buffered writes, but he was also testing on his new per-bdi write back code. If your next run could be buffered instead of
O_DIRECT, I'd be curious to see the results.
Buffered does look a lot better. I don't have a btrfs baseline before these latest changes for this exact workload, but these results are not bad at all. With cow, beats just about everything except XFS, and with nocow simply screams. CPU consumption looks good as well. I'll probably give the full set of tests a run tonight.
Results are here:
http://btrfs.boxacle.net/repository/raid/buffered-creates/buffered-create/buffered-create.html
Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:
btrfs1 kernel: [251789.525886] ------------[ cut here ]------------
Message from syslogd@ at Mon Sep 14 13:13:04 2009 ...
btrfs1 kernel: [251789.526574] invalid opcode: 0000 [#1] SMP
Message from syslogd@ at Mon Sep 14 13:13:04 2009 ...
btrfs1 kernel: [251789.526654] last sysfs file: /sys/devices/pci0000:0c/0000:0c:01.0/local_cpus
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] Stack:
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] ffff88013fc234c0 ffff88013fbcf400 0000000000000000 ffff88013fc01080
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] ffff880132e11d38 ffffffff802a5392 0000000000000001 ffff88013fbcf400
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] Call Trace:
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802a5392>] cache_flusharray+0x7d/0xae
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802a5629>] kfree+0x192/0x1b1
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffffa0378c9f>] put_worker+0x14/0x16 [btrfs]
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffffa0378d55>] btrfs_stop_workers+0xb4/0xc9 [btrfs]
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffffa0355cbe>] close_ctree+0x210/0x288 [btrfs]
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802bd1a1>] ? invalidate_inodes+0x100/0x112
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffffa033f4cb>] btrfs_put_super+0x18/0x27 [btrfs]
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802ad12b>] generic_shutdown_super+0x73/0xe2
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802ad1e5>] kill_anon_super+0x11/0x3b
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802ad51d>] deactivate_super+0x62/0x77
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802bf9eb>] mntput_no_expire+0xec/0x12c
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802bff3a>] sys_umount+0x2c5/0x31c
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff8020ba2b>] system_call_fastpath+0x16/0x1b
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] Code: 89 f7 e8 48 07 f8 ff 48 c1 e8 0c 48 ba 00 00 00 00 00 e2 ff ff 48 6b c0 38 48 01 d0 66 83 38 00 79 04 48 8b 40 10 80 38 00 78 04 <0f> 0b eb fe 48 8b 58 30 48 63 45 c8 48 89 df 4d 8b a4 c5 60 08
Steve
Quoted 3 lines
Thanks,
Chris
Quoted 36 lines · Steven Pratt
Quoted 26 lines · Chris Mason
Quoted 25 lines · Steven Pratt
Quoted 9 lines · Chris Mason
Quoted 6 lines · Steven Pratt
Better late than never. Finally got this finished up. Mixed bag on this one. BTRFS lags significantly on single threaded. Seems unable to keep IO outstanding to the device. Less that 60% busy on the DM device, compared to 97%+ for all other filesystems. nodatacow helps out, increasing utilization to about 70%, but still trails by a large margin.
Hi Steve,
Quoted 8 lines
Jens Axboe did some profiling on his big test rig and I think we found the biggest CPU problems. The end result is now setting in the master branch of the btrfs-unstable repo.
On his boxes, btrfs went from around 400MB/s streaming writes to 1GB/s limit, and we're now tied with XFS while using less CPU time.
Hopefully you will see similar results ;)
Hmmm, well no I didn't. Throughputs at 1 and 128 threads are pretty much unchanged, although I do see a good CPU savings on the 128 thread case (with cow). For 16 threads we actually regressed with cow enabled.
Results are here:
Quoted 9 lines
Quoted 4 lines
http://btrfs.boxacle.net/repository/raid/large_create_test/write-test/1M_odirect_create.html
I'll try to look more into this next week.
Hmmm, Jens was benchmarking buffered writes, but he was also testing on his new per-bdi write back code. If your next run could be buffered instead of
O_DIRECT, I'd be curious to see the results.
Quoted 6 lines
Buffered does look a lot better. I don't have a btrfs baseline before these latest changes for this exact workload, but these results are not bad at all. With cow, beats just about everything except XFS, and with nocow simply screams. CPU consumption looks good as well. I'll probably give the full set of tests a run tonight.
Wow, good news at last ;) For the oops, try the patch below (I need to push it out, but I think it'll help).
I'll try to figure out the O_DIRECT problems.
-chris
fs/btrfs/async-thread.c +2 -2
@@ -177,7 +177,7 @@ static int try_worker_shutdown(struct btrfs_worker_thread *worker)int freeit =0 ;spin_lock_irq (&worker->lock); -spin_lock_irq (&worker->workers->lock); +spin_lock (&worker->workers->lock); if (worker->workers->num_workers > 1 && worker->idle && !worker->working &&
@@ -188,7 +188,7 @@ static int try_worker_shutdown(struct btrfs_worker_thread *worker)list_del_init (&worker->worker_list); worker->workers->num_workers--; } -spin_unlock_irq (&worker->workers->lock); +spin_unlock (&worker->workers->lock);spin_unlock_irq (&worker->lock);if (freeit)
Quoted 4 lines · Steven Pratt
Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:
I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.
-chris
Quoted 11 lines · Chris Mason
Quoted 4 lines · Steven Pratt
Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:
I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.
Glad you said that. Keeps me from sending the email that said the patch didn't help :-)
Steve
Quoted 7 lines
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Steven Pratt
Quoted 7 lines · Chris Mason
Quoted 4 lines · Steven Pratt
Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:
Quoted 7 lines
Quoted 4 lines
I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.
Glad you said that. Keeps me from sending the email that said the patch didn't help :-)
Steve
Well, still getting oopses even with new code.
Lots of:
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] BUG: soft lockup - CPU#10 stuck for 61s! [btrfs-endio-1:30250] Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Pid: 30250, comm: btrfs-endio-1 Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]- Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RIP: 0010:[<ffffffff81153920>] [<ffffffff81153920>] crc32c+0x20/0x26 Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RSP: 0018:ffff88013a857cc8 EFLAGS: 00000217 Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RAX: 0000000000000040 RBX: ffff88013a857cc8 RCX: ffff88013d8022c0 Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RDX: 0000000000000010 RSI: ffff88001d349ff0 RDI: 0000000041703e71 Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RBP: ffffffff8100c4ee R08: 0000000000000000 R09: 0000000000000000 Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] R10: ffff88013a857d30 R11: 0000000000000002 R12: ffff88013a857d10 Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] R13: 0000000000000002 R14: ffff88013a857cb0 R15: ffffffff8100c38e Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] FS: 0000000000000000(0000) GS:ffff880028159000(0000) knlGS:0000000000000000 Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] CR2: 0000000000000043 CR3: 00000001368f7000 CR4: 00000000000006e0 Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Call Trace: Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffff8115397e>] ? chksum_update+0x10/0x18 Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffff81150084>] ? crypto_shash_update+0x1a/0x1c Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffff81175c34>] ? crc32c+0x4c/0x60 Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffffa0391d0f>] ? get_state_private+0x38/0x6f [btrfs] Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffffa0376688>] ? btrfs_csum_data+0xd/0xf [btrfs] Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffffa037fefc>] ? btrfs_readpage_end_io_hook+0x158/0x27b [btrfs] Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffffa0392a46>] ? end_bio_extent_readpage+0xb8/0x1c0 [btrfs] Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffff810e5733>] ? bio_endio+0x26/0x28 Sep 16 11:07:27 btrfs1 kernel: [ 1862.947656] [<ffffffffa037666e>] ? end_workqueue_fn+0x111/0x11e [btrfs] Sep 16 11:07:27 btrfs1 kernel: [ 1862.947823] [<ffffffffa039a490>] ? worker_loop+0x12a/0x3ea [btrfs] Sep 16 11:07:27 btrfs1 kernel: [ 1862.947823] [<ffffffffa039a366>] ? worker_loop+0x0/0x3ea [btrfs] Sep 16 11:07:27 btrfs1 kernel: [ 1862.948800] [<ffffffff810544e4>] ? kthread+0x8f/0x97 Sep 16 11:07:27 btrfs1 kernel: [ 1862.948800] [<ffffffff8100ca1a>] ? child_rip+0xa/0x20 Sep 16 11:07:27 btrfs1 kernel: [ 1862.948800] [<ffffffff81054455>] ? kthread+0x0/0x97 Sep 16 11:07:27 btrfs1 kernel: [ 1862.948800] [<ffffffff8100ca10>] ? child_rip+0x0/0x20
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] Pid: 31421, comm: btrfs-endio-wri Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]- Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] RIP: 0010:[<ffffffffa036afb3>] [<ffffffffa036afb3>] alloc_reserved_file_extent+0x8d/0x1c3 [btrfs] Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] RSP: 0018:ffff8800aa555af0 EFLAGS: 00010282 Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] RAX: 00000000ffffffef RBX: ffff88013b55e000 RCX: 0000000000000002 Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88012f20a9a0 Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] RBP: ffff8800aa555b60 R08: ffff8800aa555888 R09: ffff8800aa555880 Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] R10: ffff880077937400 R11: 00000000fffffffa R12: 000000000000001d Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] R13: ffff880077937400 R14: 0000000000000000 R15: 0000000000000000 Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] FS: 0000000000000000(0000) GS:ffff88002804b000(0000) knlGS:0000000000000000 Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] CR2: 00000000007c0000 CR3: 000000013e038000 CR4: 00000000000006f0 Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] Process btrfs-endio-wri (pid: 31421, threadinfo ffff8800aa554000, task ffff8801395447a0) Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] Stack: Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] ffff880077937400 0000000000000a7c 0000000000000005 0000000000000000 Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] <0> ffff880101d0c800 ffff8801140bbd20 000000b2aa555b60 ffffffffa036a190 Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] <0> 000000350000091d ffff8801090fdd40 ffff88013a4e9d40 0000000000000001 Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] Call Trace: Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa036a190>] ? update_reserved_extents+0xa7/0xbe [btrfs] Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa036f430>] run_one_delayed_ref+0x382/0x42f [btrfs] Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffff8100c4ee>] ? apic_timer_interrupt+0xe/0x20 Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa03700b1>] run_clustered_refs+0x237/0x2b4 [btrfs] Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa03a5665>] ? btrfs_find_ref_cluster+0xdc/0x115 [btrfs] Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa03701da>] btrfs_run_delayed_refs+0xac/0x195 [btrfs] Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa0379a76>] __btrfs_end_transaction+0x59/0xfe [btrfs] Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa0379b36>] btrfs_end_transaction+0xb/0xd [btrfs] Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa037f29b>] btrfs_finish_ordered_io+0x23c/0x265 [btrfs] Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa037f2d9>] btrfs_writepage_end_io_hook+0x15/0x17 [btrfs] Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa0392901>] end_bio_extent_writepage+0xa5/0x132 [btrfs] Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffff810e5733>] bio_endio+0x26/0x28 Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa037666e>] end_workqueue_fn+0x111/0x11e [btrfs] Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa039a490>] worker_loop+0x12a/0x3ea [btrfs] Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa039a366>] ? worker_loop+0x0/0x3ea [btrfs] Sep 16 11:54:48 btrfs1 kernel: [ 4703.082621] [<ffffffff810544e4>] kthread+0x8f/0x97 Sep 16 11:54:48 btrfs1 kernel: [ 4703.082621] [<ffffffff8100ca1a>] child_rip+0xa/0x20 Sep 16 11:54:48 btrfs1 kernel: [ 4703.082621] [<ffffffff81054455>] ? kthread+0x0/0x97 Sep 16 11:54:48 btrfs1 kernel: [ 4703.082621] [<ffffffff8100ca10>] ? child_rip+0x0/0x20 Sep 16 11:54:48 btrfs1 kernel: [ 4703.082621] Code: 08 4c 8d 45 d4 41 8d 44 24 18 48 8b 73 20 48 8b 4d 18 41 b9 01 00 00 00 48 8b 7d b8 4c 89 ea
89 45 d4 e8 93 e3 ff ff 85 c0 74 04 <0f> 0b eb fe 49 63 75 40 4d 8b 65 00 49 83 cf 01 4c 89 e7 48 6b
Happened on 2 machines.
Steve
Quoted 13 lines
Quoted 8 lines
-chris
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlSignature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 16 lines · Steven Pratt
Quoted 6 lines · Steven Pratt
Quoted 5 lines · Chris Mason
Quoted 3 lines · Steven Pratt
Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:
Quoted 7 lines
Quoted 3 lines
I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.
Glad you said that. Keeps me from sending the email that said the patch didn't help :-)
Steve
Well, still getting oopses even with new code.
Quoted 7 lines
Lots of:
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] BUG: soft lockup - CPU#10 stuck for 61s! [btrfs-endio-1:30250] Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Pid: 30250, comm: btrfs-endio-1 Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]- Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RIP:0010:[<ffffffff81153920>] [<
ffffffff81153920>]crc32c+0x20/0x26
If I'm reading this right, you've got a softlockup in crc32c? Something has gone really wrong here. Are you reusing datasets from old runs?
-chris
Quoted 32 lines · Chris Mason
Quoted 21 lines · Steven Pratt
Quoted 8 lines · Steven Pratt
Quoted 6 lines · Chris Mason
Quoted 4 lines · Steven Pratt
Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:
Quoted 9 lines
Quoted 4 lines
I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.
Glad you said that. Keeps me from sending the email that said the patch didn't help :-)
Steve
Well, still getting oopses even with new code.
Quoted 8 lines
Lots of:
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] BUG: soft lockup - CPU#10 stuck for 61s! [btrfs-endio-1:30250] Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Pid: 30250, comm: btrfs-endio-1 Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]- Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RIP:0010:[<ffffffff81153920>] [<
ffffffff81153920>]crc32c+0x20/0x26
Quoted 3 lines
If I'm reading this right, you've got a softlockup in crc32c? Something has gone really wrong here. Are you reusing datasets from old runs?
No, mkfs before every run.
Steve
Quoted 6 lines
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 32 lines · Chris Mason
Quoted 21 lines · Steven Pratt
Quoted 8 lines · Steven Pratt
Quoted 6 lines · Chris Mason
Quoted 4 lines · Steven Pratt
Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:
Quoted 9 lines
Quoted 4 lines
I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.
Glad you said that. Keeps me from sending the email that said the patch didn't help :-)
Steve
Well, still getting oopses even with new code.
Quoted 8 lines
Lots of:
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] BUG: soft lockup - CPU#10 stuck for 61s! [btrfs-endio-1:30250] Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Pid: 30250, comm: btrfs-endio-1 Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]- Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RIP:0010:[<ffffffff81153920>] [<
ffffffff81153920>]crc32c+0x20/0x26
Quoted 3 lines
If I'm reading this right, you've got a softlockup in crc32c? Something has gone really wrong here. Are you reusing datasets from old runs?
From the second machine a single bug:
Sep 16 11:53:42 btrfs2 kernel: [ 3769.298240] ------------[ cut here ]------------ Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] kernel BUG at fs/btrfs/extent-tree.c:4097! Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] invalid opcode: 0000 [#1] SMP Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] last sysfs file: /sys/devices/system/cpu/cpu15/cache/index1/shared_cpu_map Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] CPU 9 Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler btrfs zlib_deflate oprofile autofs4 nfs lockd nfs_acl auth_rpc gss sunrpc dm_multipath video output sbs sbshc battery ac parport_pc lp parport sg joydev serio_raw acpi_memhotplug rtc_cmos rtc_core rtc_lib button tg3 libphy i2c_ piix4 i2c_core pcspkr dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod lpfc scsi_transport_fc aic94xx libsas libata scsi_transport_sas sd_mod scsi_mod ext 3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode] Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] Pid: 2106, comm: btrfs-endio-wri Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]- Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] RIP: 0010:[<ffffffffa0386fb3>] [<ffffffffa0386fb3>] alloc_reserved_file_extent+0x8d/0x1c3 [btrfs] Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] RSP: 0018:ffff88002758faf0 EFLAGS: 00010282 Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] RAX: 00000000ffffffef RBX: ffff880136434000 RCX: 0000000000000002 Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8800a7040370 Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] RBP: ffff88002758fb60 R08: ffff88002758f958 R09: ffff88002758f950 Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] R10: 0000000000000004 R11: ffff8800a7040370 R12: 000000000000001d Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] R13: ffff8800b79e6910 R14: 0000000000000000 R15: 0000000000000000 Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] FS: 0000000000000000(0000) GS:ffff88002813e000(0000) knlGS:0000000000000000 Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] CR2: 00007f1f6915a000 CR3: 000000013dd4e000 CR4: 00000000000006e0 Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] Process btrfs-endio-wri (pid: 2106, threadinfo ffff88002758e000, task ffff88013b94c100) Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] Stack: Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] ffff8800709fc760 0000000000000856 0000000000000005 0000000000000000 Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] <0> ffff8801329d5000 ffff880102242de0 000000b22758fb60 ffffffffa0386190 Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] <0> 00000035329d5000 ffff880128291440 ffff880108302340 0000000000000001 Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] Call Trace: Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffffa0386190>] ? update_reserved_extents+0xa7/0xbe [btrfs] Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffffa038b430>] run_one_delayed_ref+0x382/0x42f [btrfs] Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffffa038c0b1>] run_clustered_refs+0x237/0x2b4 [btrfs] Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffffa03c1665>] ? btrfs_find_ref_cluster+0xdc/0x115 [btrfs] Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffffa038c1da>] btrfs_run_delayed_refs+0xac/0x195 [btrfs] Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffffa0395a76>] __btrfs_end_transaction+0x59/0xfe [btrfs] Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffffa0395b36>] btrfs_end_transaction+0xb/0xd [btrfs] Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffffa039b29b>] btrfs_finish_ordered_io+0x23c/0x265 [btrfs] Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffffa039b2d9>] btrfs_writepage_end_io_hook+0x15/0x17 [btrfs] Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffffa03ae901>] end_bio_extent_writepage+0xa5/0x132 [btrfs] Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffff810e5733>] bio_endio+0x26/0x28 Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffffa039266e>] end_workqueue_fn+0x111/0x11e [btrfs] Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffffa03b6490>] worker_loop+0x12a/0x3ea [btrfs] Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffffa03b6366>] ? worker_loop+0x0/0x3ea [btrfs] Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffff810544e4>] kthread+0x8f/0x97 Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffff8100ca1a>] child_rip+0xa/0x20 Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffff81054455>] ? kthread+0x0/0x97 Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] [<ffffffff8100ca10>] ? child_rip+0x0/0x20 Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] Code: 08 4c 8d 45 d4 41 8d 44 24 18 48 8b 73 20 48 8b 4d 18 41 b9 01 00 00 00 48 8b 7d b8 4c 89 ea 89 45 d4 e8 93 e3 f
f ff 85 c0 74 04 <0f> 0b eb fe 49 63 75 40 4d 8b 65 00 49 83 cf 01 4c 89 e7 48 6b
Steve
Quoted 6 lines
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 25 lines · Steven Pratt
Quoted 24 lines · Chris Mason
Quoted 22 lines · Steven Pratt
Quoted 5 lines · Steven Pratt
Quoted 4 lines · Chris Mason
Quoted 3 lines · Steven Pratt
Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:
Quoted 7 lines
Quoted 3 lines
I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.
Glad you said that. Keeps me from sending the email that said the patch didn't help :-)
Steve
Well, still getting oopses even with new code.
Lots of:
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] BUG: soft lockup - CPU#10 stuck for 61s! [btrfs-endio-1:30250] Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Pid: 30250, comm: btrfs-endio-1 Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]- Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RIP:0010:[<ffffffff81153920>] [<
ffffffff81153920>]crc32c+0x20/0x26
Quoted 3 lines
Quoted 2 lines
If I'm reading this right, you've got a softlockup in crc32c? Something has gone really wrong here. Are you reusing datasets from old runs?
No, mkfs before every run.
Could you please send me the full softlockup output? Its hard to read all line wrapped, so the original files would help.
-chris
Quoted 25 lines · Steven Pratt
Quoted 24 lines · Chris Mason
Quoted 22 lines · Steven Pratt
Quoted 5 lines · Steven Pratt
Quoted 4 lines · Chris Mason
Quoted 3 lines · Steven Pratt
Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:
Quoted 7 lines
Quoted 3 lines
I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.
Glad you said that. Keeps me from sending the email that said the patch didn't help :-)
Steve
Well, still getting oopses even with new code.
Lots of:
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] BUG: soft lockup - CPU#10 stuck for 61s! [btrfs-endio-1:30250] Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Pid: 30250, comm: btrfs-endio-1 Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]- Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RIP:0010:[<ffffffff81153920>] [<
ffffffff81153920>]crc32c+0x20/0x26
Quoted 4 lines
Quoted 2 lines
If I'm reading this right, you've got a softlockup in crc32c? Something has gone really wrong here. Are you reusing datasets from old runs?
From the second machine a single bug:
Sep 16 11:53:42 btrfs2 kernel: [ 3769.298240] ------------[ cut here
Ok, which mount options and job file is this from?
-chris
Quoted 35 lines · Chris Mason
Quoted 33 lines · Steven Pratt
Quoted 31 lines · Chris Mason
Quoted 29 lines · Steven Pratt
Quoted 8 lines · Steven Pratt
Quoted 6 lines · Chris Mason
Quoted 4 lines · Steven Pratt
Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:
Quoted 9 lines
Quoted 4 lines
I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.
Glad you said that. Keeps me from sending the email that said the patch didn't help :-)
Steve
Well, still getting oopses even with new code.
Lots of:
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] BUG: soft lockup - CPU#10 stuck for 61s! [btrfs-endio-1:30250] Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Pid: 30250, comm: btrfs-endio-1 Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]- Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RIP:0010:[<ffffffff81153920>] [<
ffffffff81153920>]crc32c+0x20/0x26
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
If I'm reading this right, you've got a softlockup in crc32c? Something has gone really wrong here. Are you reusing datasets from old runs?
From the second machine a single bug:
Sep 16 11:53:42 btrfs2 kernel: [ 3769.298240] ------------[ cut hereOk, which mount options and job file is this from?
mount -t btrfs /dev/ffsbdev1 /mnt/ffsb1'
[20090916-11:47:37.738883526] PROCESSING COMMAND : 'run random_writes__threads_0001 ffsb http://hks.austin.ibm.com/users/corry/btrfs/ffsb/profiles/btrfs2/random_writes.ffsb num_threads=1'
So , this is single disk machine, running single threaded random write workload. Buffered, not odirect.
I'm packaging up full messages file, repeated errros makes it big. Will send separately.
Steve
Quoted 6 lines
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Chris Mason
Quoted 3 lines · Steven Pratt
Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:
I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.
Chris:
FYI - all five of my test systems have now finished my standard test cycle on the -unstable master branch, and I've not seen a single hang. So, your fix for the async thread shutdown race seems to have fixed my problems, even if Steve's still seeing trouble.
I'll note that the running times for fsstress on some of my systems have become rather longer with btrfs-unstable/master kernels - 3.5 rather than 2.5 hours on multidevice filesystems. Running times on single device filesystems are roughly the same.
I'm going to start another set of tests for thoroughness unless you've got more patches coming.
Thanks,
Eric
Quoted 3 lines
-chris
Quoted 12 lines · Eric Whitney
Quoted 8 lines · Chris Mason
Quoted 3 lines · Steven Pratt
Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:
I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.
Quoted 7 lines
Chris:
FYI - all five of my test systems have now finished my standard test cycle on the -unstable master branch, and I've not seen a single hang. So, your fix for the async thread shutdown race seems to have fixed my problems, even if Steve's still seeing trouble.
I'll note that the running times for fsstress on some of my systems have become rather longer with btrfs-unstable/master kernels - 3.5 rather than 2.5 hours on multidevice filesystems. Running times on single device filesystems are roughly the same.
I'm going to start another set of tests for thoroughness unless you've got more patches coming.
I've had some offline discussions with Chris, and it seems the problem is triggered by unmounting and re-mounting the file system between tests (but not running mkfs again). I have also just verified that the problem does not occur if repeated tests are run without the unmount mount cycle. So in case this is not clear:
mkfs
mount
create new files
run test
umount
mount
delete old files
create new files
run test
BUG
but..
mkfs
mount
create new files
run test
umount
mkfs <------ differnet
mount
delete old files
create new files
run test
...
all is fine
or...
mkfs
mount
create new files
run test
# no mounts or mkfs here delete old files create new files run test ... all is fine
Steve
Quoted 7 lines
Thanks,
EricQuoted 3 lines
-chris
Quoted 25 lines · Steven Pratt
Quoted 24 lines · Eric Whitney
Quoted 8 lines · Chris Mason
Quoted 3 lines · Steven Pratt
Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:
I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.
Chris:
FYI - all five of my test systems have now finished my standard test cycle on the -unstable master branch, and I've not seen a single hang. So, your fix for the async thread shutdown race seems to have fixed my problems, even if Steve's still seeing trouble.
I'll note that the running times for fsstress on some of my systems have become rather longer with btrfs-unstable/master kernels - 3.5 rather than 2.5 hours on multidevice filesystems. Running times on single device filesystems are roughly the same.
Quoted 7 lines
Quoted 2 lines
I'm going to start another set of tests for thoroughness unless you've got more patches coming.
I've had some offline discussions with Chris, and it seems the problem is triggered by unmounting and re-mounting the file system
between tests (but not running mkfs again). I have also just
verified that the problem does not occur if repeated tests are run without the unmount mount cycle. So in case this is not clear:
Ok, I've triggered it here. Next step is trying Yan Zheng's async caching update.
------------[ cut here ]------------ kernel BUG at fs/btrfs/extent-tree.c:4097!
invalid opcode: 0000 [#1] SMP
-chris
[ crashes on runs involving unmounts ]
The run is still going here, but it has survived longer than before. I'm trying with Yan Zheng's patch:
From: Yan Zheng <[email protected]>
Date: Fri, 11 Sep 2009 16:11:19 -0400
Subject: [PATCH] Btrfs: improve async block group caching
This patch gets rid of two limitations of async block group caching. The old code delays handling pinned extents when block group is in caching. To allocate logged file extents, the old code need wait until block group is fully cached. To get rid of the limitations, This patch introduces a data structure to track the progress of caching. Base on the caching progress, we know which extents should be added to the free space cache when handling the pinned extents. The logged file extents are also handled in a similar way.
This patch also changes how pinned extents are tracked. The old code uses one tree to track pinned extents, and copy the pinned extents tree at transaction commit time. This patch makes it use two trees to track pinned extents. One tree for extents that are pinned in the running transaction, one tree for extents that can be unpinned. At transaction commit time, we swap the two trees.
--- fs/btrfs/ctree.h | 29 ++- fs/btrfs/disk-io.c | 7 +- fs/btrfs/extent-tree.c | 586 +++++++++++++++++++++++++++++------------------- fs/btrfs/transaction.c | 15 +- fs/btrfs/tree-log.c | 4 +- 5 files changed, 382 insertions(+), 259 deletions(-)
fs/btrfs/ctree.h +20 -9
@@ -726,6 +726,15 @@ enum btrfs_caching_type { BTRFS_CACHE_FINISHED = 2, }; +struct btrfs_caching_control { +struct list_head list; +struct mutex mutex; +wait_queue_head_t wait; +struct btrfs_block_group_cache *block_group; +u64 progress; +atomic_t count; +}; +struct btrfs_block_group_cache {struct btrfs_key key;struct btrfs_block_group_item item;
@@ -742,8 +751,9 @@ struct btrfs_block_group_cache {int dirty;/* cache tracking stuff */ -wait_queue_head_t caching_q;int cached; +struct btrfs_caching_control *caching_ctl; +u64 last_byte_to_unpin;struct btrfs_space_info *space_info;
@@ -788,7 +798,8 @@ struct btrfs_fs_info {spinlock_t block_group_cache_lock;struct rb_root block_group_cache_tree; -struct extent_io_tree pinned_extents; +struct extent_io_tree freed_extents[2 ]; +struct extent_io_tree *pinned_extents;/* logical->physical extent mapping */ struct btrfs_mapping_tree mapping_tree;
@@ -825,8 +836,6 @@ struct btrfs_fs_info {struct mutex drop_mutex;struct mutex volume_mutex;struct mutex tree_reloc_mutex; -struct rw_semaphore extent_commit_sem; -/* * this protects the ordered operations list only while we are * processing all of the entries on it. This way we make
@@ -835,10 +844,12 @@ struct btrfs_fs_info {* before jumping into the main commit. */ struct mutex ordered_operations_mutex; +struct rw_semaphore extent_commit_sem;struct list_head trans_list;struct list_head hashers;struct list_head dead_roots; +struct list_head caching_block_groups;atomic_t nr_async_submits;atomic_t async_submit_draining;
@@ -1920,8 +1931,8 @@ void btrfs_put_block_group(struct btrfs_block_group_cache *cache);int btrfs_run_delayed_refs (struct btrfs_trans_handle *trans,struct btrfs_root *root,unsigned long count);int btrfs_lookup_extent (struct btrfs_root *root,u64 start,u64 len); -int btrfs_update_pinned_extents (struct btrfs_root *root, -u64 bytenr,u64 num,int pin); +int btrfs_pin_extent (struct btrfs_root *root, +u64 bytenr,u64 num,int reserved);int btrfs_drop_leaf_ref (struct btrfs_trans_handle *trans,struct btrfs_root *root,struct extent_buffer *leaf); int btrfs_cross_ref_exist(struct btrfs_trans_handle *trans,
@@ -1971,9 +1982,10 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans, u64 root_objectid, u64 owner, u64 offset);int btrfs_free_reserved_extent (struct btrfs_root *root,u64 start,u64 len); +int btrfs_prepare_extent_commit (struct btrfs_trans_handle *trans, +struct btrfs_root *root);int btrfs_finish_extent_commit (struct btrfs_trans_handle *trans, -struct btrfs_root *root, -struct extent_io_tree *unpin); +struct btrfs_root *root); int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans, struct btrfs_root *root, u64 bytenr, u64 num_bytes, u64 parent,
@@ -2006,7 +2018,6 @@ void btrfs_delalloc_reserve_space(struct btrfs_root *root, struct inode *inode, u64 bytes);void btrfs_delalloc_free_space (struct btrfs_root *root,struct inode *inode,u64 bytes); -void btrfs_free_pinned_extents (struct btrfs_fs_info *info);/* ctree.c */ int btrfs_bin_search (struct extent_buffer *eb,struct btrfs_key *key,int level,int *slot);
fs/btrfs/disk-io.c +5 -2
@@ -1563,6 +1563,7 @@ struct btrfs_root *open_ctree(struct super_block *sb,INIT_LIST_HEAD (&fs_info->hashers);INIT_LIST_HEAD (&fs_info->delalloc_inodes);INIT_LIST_HEAD (&fs_info->ordered_operations); +INIT_LIST_HEAD (&fs_info->caching_block_groups);spin_lock_init (&fs_info->delalloc_lock);spin_lock_init (&fs_info->new_trans_lock);spin_lock_init (&fs_info->ref_cache_lock);
@@ -1621,8 +1622,11 @@ struct btrfs_root *open_ctree(struct super_block *sb,spin_lock_init (&fs_info->block_group_cache_lock); fs_info->block_group_cache_tree.rb_node =NULL ; -extent_io_tree_init (&fs_info->pinned_extents, +extent_io_tree_init (&fs_info->freed_extents[0 ], fs_info->btree_inode->i_mapping,GFP_NOFS ); +extent_io_tree_init (&fs_info->freed_extents[1 ], + fs_info->btree_inode->i_mapping,GFP_NOFS ); + fs_info->pinned_extents = &fs_info->freed_extents[0 ]; fs_info->do_barriers =1 ;BTRFS_I (fs_info->btree_inode)->root = tree_root;
@@ -2359,7 +2363,6 @@ int close_ctree(struct btrfs_root *root)free_extent_buffer (root->fs_info->csum_root->commit_root);btrfs_free_block_groups (root->fs_info); -btrfs_free_pinned_extents (root->fs_info);del_fs_roots (fs_info);
fs/btrfs/extent-tree.c +352 -234
941 lines not on this page · raw message
fs/btrfs/transaction.c +3 -12
@@ -874,7 +874,6 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,unsigned long timeout =1 ;struct btrfs_transaction *cur_trans;struct btrfs_transaction *prev_trans =NULL ; -struct extent_io_tree *pinned_copy;DEFINE_WAIT (wait);int ret;int should_grow =0 ;
@@ -915,13 +914,6 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,return 0 ; } - pinned_copy =kmalloc (sizeof (*pinned_copy),GFP_NOFS ); -if (!pinned_copy) -return -ENOMEM ; - -extent_io_tree_init (pinned_copy, - root->fs_info->btree_inode->i_mapping,GFP_NOFS ); - trans->transaction->in_commit =1 ; trans->transaction->blocked =1 ;if (cur_trans->list.prev != &root->fs_info->trans_list) {
@@ -1019,6 +1011,8 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans, ret =commit_cowonly_roots (trans, root);BUG_ON (ret); +btrfs_prepare_extent_commit (trans, root); + cur_trans = root->fs_info->running_transaction;spin_lock (&root->fs_info->new_trans_lock); root->fs_info->running_transaction =NULL ;
@@ -1042,8 +1036,6 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,memcpy (&root->fs_info->super_for_commit, &root->fs_info->super_copy,sizeof (root->fs_info->super_copy)); -btrfs_copy_pinned (root, pinned_copy); - trans->transaction->blocked =0 ;wake_up (&root->fs_info->transaction_wait);
@@ -1059,8 +1051,7 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,*/ mutex_unlock (&root->fs_info->tree_log_mutex); -btrfs_finish_extent_commit (trans, root, pinned_copy); -kfree (pinned_copy); +btrfs_finish_extent_commit (trans, root);/* do the directory inserts of any pending snapshot creations */ finish_pending_snapshots (trans, root->fs_info);
fs/btrfs/tree-log.c +2 -2
@@ -263,8 +263,8 @@ static int process_one_buffer(struct btrfs_root *log, struct walk_control *wc, u64 gen) {if (wc->pin) -btrfs_update_pinned_extents (log->fs_info->extent_root, - eb->start, eb->len,1 ); +btrfs_pin_extent (log->fs_info->extent_root, + eb->start, eb->len,0 );if (btrfs_buffer_uptodate (eb, gen)) {if (wc->write)
Signature · 1 line
1.6.4.1
Quoted 8 lines · Chris Mason
[ crashes on runs involving unmounts ]
The run is still going here, but it has survived longer than before. I'm trying with Yan Zheng's patch:
From: Yan Zheng <[email protected]>
Date: Fri, 11 Sep 2009 16:11:19 -0400
Subject: [PATCH] Btrfs: improve async block group caching
Quick update, I got through a full run of Steve's test with this applied. I'll start a few more ;)
-chris
Quoted 7 lines · Chris Mason
Quoted 5 lines · Chris Mason
[ crashes on runs involving unmounts ]
The run is still going here, but it has survived longer than before. I'm trying with Yan Zheng's patch:
Quoted 8 lines
Quoted 4 lines
From: Yan Zheng <[email protected]>
Date: Fri, 11 Sep 2009 16:11:19 -0400
Subject: [PATCH] Btrfs: improve async block group cachingQuick update, I got through a full run of Steve's test with this applied. I'll start a few more ;)
Seems to work for me too! Got through all the random write tests with no problems. Will kick off full run overnight.
Steve
Quoted 6 lines
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Steven Pratt
Quoted 6 lines · Chris Mason
Quoted 5 lines · Chris Mason
[ crashes on runs involving unmounts ]
The run is still going here, but it has survived longer than before. I'm trying with Yan Zheng's patch:
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
From: Yan Zheng <[email protected]>
Date: Fri, 11 Sep 2009 16:11:19 -0400
Subject: [PATCH] Btrfs: improve async block group cachingQuick update, I got through a full run of Steve's test with this applied. I'll start a few more ;)
Seems to work for me too! Got through all the random write tests with no problems. Will kick off full run overnight.
Thanks again.
I've updated the btrfs-unstable tree to include Yan Zheng's fix. I've also included two more buffered writeback speedups, and I expect these to make a difference on the multi-threaded tests.
I took a look at 1MB O_DIRECT writes, and the latencies of sending off checksumming to the checksum threads seem to be the biggest problem. I get full tput at 8MB O_DIRECT writes, so for now I'm going to leave this one alone.
-chris
Quoted 24 lines · Chris Mason
Quoted 13 lines · Steven Pratt
Quoted 11 lines · Chris Mason
Quoted 9 lines · Chris Mason
[ crashes on runs involving unmounts ]
The run is still going here, but it has survived longer than before. I'm trying with Yan Zheng's patch:
From: Yan Zheng <[email protected]>
Date: Fri, 11 Sep 2009 16:11:19 -0400
Subject: [PATCH] Btrfs: improve async block group cachingQuoted 6 lines
Quoted 3 lines
Quick update, I got through a full run of Steve's test with this applied. I'll start a few more ;)
Seems to work for me too! Got through all the random write tests with no problems. Will kick off full run overnight.
Thanks again.
Quoted 9 lines
I've updated the btrfs-unstable tree to include Yan Zheng's fix. I've also included two more buffered writeback speedups, and I expect these to make a difference on the multi-threaded tests.
I took a look at 1MB
O_DIRECTwrites, and the latencies of sending off checksumming to the checksum threads seem to be the biggest problem. I get full tput at 8MBO_DIRECTwrites, so for now I'm going to leave this one alone.
Updated performance results are available. Ran both the 9/16 tree + async patches and the tree from 9/18. Results are a mixed bag, some faster some slower.
http://btrfs.boxacle.net/repository/raid/history/History.html
Steve
Quoted 7 lines
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html