Skip to contentSkip to conversation

Updated performance results

DISCUSSION Steven Pratt 6 participants 38 messages Jul 23, 2009 – Sep 23, 2009 Btrfs file system

lore
Share & tools
lore
#0 lore raw

I have re-run the raid tests with re-creating the fileset between each of the random write workloads and performance does now match the previous newformat results. The bad news is that the huge gain that I had attributed to the newformat release, does not really exist. All of the previous results(except for the newformat run) were not re-creating the fileset, so the gain in performance was due only to having a fresh set of files, not any code changes.

So, I have done 2 new sets of runs to look into this further. One is a 3 hour run of single threaded random write to the RAID system. I have compared this to ext3. Performance results are here: http://btrfs.boxacle.net/repository/raid/longwrite/longwrite/Longrandomwrite.html

and graphing of all the iostat data can be found here:

http://btrfs.boxacle.net/repository/raid/longwrite/summary.html

The iostat graphs for btrfs are interesting for a number of reasons. First, it takes about 3000 seconds (or 50 minutes) for btrfs to reach steady state. Second, if you look at write throughput from the device view vs. the btrfs/application view, we see that for a application throughput of 21.5MB/sec it requires 63MB/sec of actual disk writes. That is an overhead of 3 to 1 vs an overhead of ~0 for ext3. Also, looking at the change in iops vs MB/sec, we see that while btrfs starts out with reasonable size IOs, it quickly deteriorate to an average IO size of only 13kb. Remember, the starting file set is only 100GB on a 2.1TB filesystem, and all data is overwrite, and this is single threaded, so there is no reason this should fragment. It seems like the allocator is having a problem doing sequential allocations.

Another set of runs I did was to do repetitive 5 minute random write runs. Results are here:
http://btrfs.boxacle.net/repository/raid/repeat/repeat/repeat.html

This shows the dramatic degredation after just a short time, but I believe there is a fair amount of overhead in btrfs after newly mounting the FS (which this test did between each run) so I repeated without unmounting and remounting and results are here:
http://btrfs.boxacle.net/repository/raid/repeat-nomount/repeat/repeat-nomount.html

These results show a much less dramatic degradation, but btrfs still degrades by over 40% in just 30 minutes of run time. In fact, it was still degrading by 10% every 5 minutes when this test ended.

Steve

#1 lore raw
Quoted 7 lines · Steven Pratt

I have re-run the raid tests with re-creating the fileset between each of the random write workloads and performance does now match the previous newformat results. The bad news is that the huge gain that I had attributed to the newformat release, does not really exist. All of the previous results(except for the newformat run) were not re-creating the fileset, so the gain in performance was due only to having a fresh set of files, not any code changes.

Thanks for doing all of these runs. This is still a little different than what I have here, my initial runs are very very fast and after 10 or so level out to a relatively low performance on random writes. With nodatacow, it stays even.

Quoted 10 lines

So, I have done 2 new sets of runs to look into this further. One is a 3 hour run of single threaded random write to the RAID system. I have compared this to ext3. Performance results are here:
http://btrfs.boxacle.net/repository/raid/longwrite/longwrite/Longrandomwrite.html

and graphing of all the iostat data can be found here:

http://btrfs.boxacle.net/repository/raid/longwrite/summary.html

Quoted 12 lines

The iostat graphs for btrfs are interesting for a number of reasons. First, it takes about 3000 seconds (or 50 minutes) for btrfs to reach steady state. Second, if you look at write throughput from the device view vs. the btrfs/application view, we see that for a application throughput of 21.5MB/sec it requires 63MB/sec of actual disk writes. That is an overhead of 3 to 1 vs an overhead of ~0 for ext3. Also, looking at the change in iops vs MB/sec, we see that while btrfs starts out with reasonable size IOs, it quickly deteriorate to an average IO size of only 13kb. Remember, the starting file set is only 100GB on a 2.1TB filesystem, and all data is overwrite, and this is single threaded, so there is no reason this should fragment. It seems like the allocator is having a problem doing sequential allocations.

There are two things happening. First the default allocation scheme isn't very well suited to this, mount -o ssd will perform better. But over the long term, random overwrites to the file cause a lot of writes to the extent allocation tree. That's really what -o nodatacow is saving us. There are optimizations we can do, but we're holding off on that in favor of enospc and other pressing things.

But, with all of that said, Josef has some really important allocator improvements. I've put them out along with our pending patches into the experimental branch of the btrfs-unstable tree. Could you please give this branch a try both with and without the ssd mount option?

-chris

#2 lore raw
Quoted 11 lines · Chris Mason
Quoted 2 lines · Steven Pratt

I have re-run the raid tests with re-creating the fileset between each of the random write workloads and performance does now match the previous newformat results. The bad news is that the huge gain that I had attributed to the newformat release, does not really exist. All of the previous results(except for the newformat run) were not re-creating the fileset, so the gain in performance was due only to having a fresh set of files, not any code changes.

Thanks for doing all of these runs. This is still a little different than what I have here, my initial runs are very very fast and after 10 or so level out to a relatively low performance on random writes. With nodatacow, it stays even.

Right, I do not see this problem with nodatacow.

Quoted 9 lines
Quoted 8 lines

So, I have done 2 new sets of runs to look into this further. One is a 3 hour run of single threaded random write to the RAID system. I have compared this to ext3. Performance results are here: http://btrfs.boxacle.net/repository/raid/longwrite/longwrite/Longrandomwrite.html

and graphing of all the iostat data can be found here:

http://btrfs.boxacle.net/repository/raid/longwrite/summary.html

The iostat graphs for btrfs are interesting for a number of reasons. First, it takes about 3000 seconds (or 50 minutes) for btrfs to reach steady state. Second, if you look at write throughput from the device view vs. the btrfs/application view, we see that for a application throughput of 21.5MB/sec it requires 63MB/sec of actual disk writes. That is an overhead of 3 to 1 vs an overhead of ~0 for ext3. Also, looking at the change in iops vs MB/sec, we see that while btrfs starts out with reasonable size IOs, it quickly deteriorate to an average IO size of only 13kb. Remember, the starting file set is only 100GB on a 2.1TB filesystem, and all data is overwrite, and this is single threaded, so there is no reason this should fragment. It seems like the allocator is having a problem doing sequential allocations.

Quoted 7 lines

There are two things happening. First the default allocation scheme isn't very well suited to this, mount -o ssd will perform better. But over the long term, random overwrites to the file cause a lot of writes to the extent allocation tree. That's really what -o nodatacow is saving us. There are optimizations we can do, but we're holding off on that in favor of enospc and other pressing things.

Well I have -o ssd data that I can upload, but it was worse than without. I do understand about timing and priorities.

Quoted 6 lines

But, with all of that said, Josef has some really important allocator improvements. I've put them out along with our pending patches into the experimental branch of the btrfs-unstable tree. Could you please give this branch a try both with and without the ssd mount option?

Sure, will try to get to it tomorrow.

Steve

Quoted 3 lines

-chris

#3 lore raw
Quoted 54 lines · Steven Pratt
Quoted 11 lines · Chris Mason
Quoted 8 lines · Steven Pratt

I have re-run the raid tests with re-creating the fileset between each of the random write workloads and performance does now match the previous newformat results. The bad news is that the huge gain that I had attributed to the newformat release, does not really exist. All of the previous results(except for the newformat run) were not re-creating the fileset, so the gain in performance was due only to having a fresh set of files, not any code changes.

Quoted 6 lines

Thanks for doing all of these runs. This is still a little different than what I have here, my initial runs are very very fast and after 10 or so level out to a relatively low performance on random writes. With nodatacow, it stays even.

Right, I do not see this problem with nodatacow.

Quoted 24 lines
Quoted 9 lines

So, I have done 2 new sets of runs to look into this further. One is a 3 hour run of single threaded random write to the RAID system. I have compared this to ext3. Performance results are here:
http://btrfs.boxacle.net/repository/raid/longwrite/longwrite/Longrandomwrite.html

and graphing of all the iostat data can be found here:

http://btrfs.boxacle.net/repository/raid/longwrite/summary.html

Quoted 14 lines

The iostat graphs for btrfs are interesting for a number of reasons.
First, it takes about 3000 seconds (or 50 minutes) for btrfs to
reach steady state. Second, if you look at write throughput from the device view vs. the btrfs/application view, we see that for a application throughput of 21.5MB/sec it requires 63MB/sec of actual
disk writes. That is an overhead of 3 to 1 vs an overhead of ~0 for
ext3. Also, looking at the change in iops vs MB/sec, we see that while btrfs starts out with reasonable size IOs, it quickly deteriorate to an average IO size of only 13kb. Remember, the starting file set is only 100GB on a 2.1TB filesystem, and all data is overwrite, and this is single threaded, so there is no reason this should fragment. It seems like the allocator is having a problem doing sequential allocations.

Quoted 7 lines

There are two things happening. First the default allocation scheme isn't very well suited to this, mount -o ssd will perform better. But over the long term, random overwrites to the file cause a lot of writes to the extent allocation tree. That's really what -o nodatacow is saving us. There are optimizations we can do, but we're holding off on that in favor of enospc and other pressing things.

Well I have -o ssd data that I can upload, but it was worse than without. I do understand about timing and priorities.

Quoted 7 lines
Quoted 6 lines

But, with all of that said, Josef has some really important allocator improvements. I've put them out along with our pending patches into the experimental branch of the btrfs-unstable tree. Could you please give this branch a try both with and without the ssd mount option?

Sure, will try to get to it tomorrow.

Sorry, I missed a fix in the experimental branch. I'll push out a rebased version in a few minutes.

-chris

#4 lore raw
Quoted 6 lines · Chris Mason
Quoted 2 lines
Quoted 1 line

Sure, will try to get to it tomorrow.

Sorry, I missed a fix in the experimental branch. I'll push out a rebased version in a few minutes.

Ok, the rebased version is ready to use.

-chris

#5 lore raw
Quoted 4 lines · Chris Mason
Quoted 2 lines · Chris Mason
Quoted 2 lines
Quoted 2 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

Sure, will try to get to it tomorrow.

Sorry, I missed a fix in the experimental branch. I'll push out a rebased version in a few minutes.

Ok, the rebased version is ready to use.

Ok, good. Also seems I misspoke on the -o ssd results. I looked them over again this morning and they are slightly bette than without. Initial score of 46MB/sec vs 44MB/sec without, and degrades after 30 minutes to about 25MB/sec vs 20MB/sec without. So after 30 minutes of runtime -o ssd is running about 25% faster, but still significant degrade.

Steve

Quoted 7 lines

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
In reply to Chris Mason
Quoted 4 lines · Chris Mason
Quoted 2 lines · Chris Mason
Quoted 2 lines
Quoted 2 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

Sure, will try to get to it tomorrow.

Sorry, I missed a fix in the experimental branch. I'll push out a rebased version in a few minutes.

Ok, the rebased version is ready to use.

New results are up for both with and without nodatacow. Not much change.

http://btrfs.boxacle.net/repository/raid/history/History.html

Have another run going with nodatacow and ssd.

Steve

Quoted 7 lines

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw
Quoted 7 lines · Steven Pratt
Quoted 6 lines · Chris Mason
Quoted 4 lines · Chris Mason
Quoted 1 line

Sure, will try to get to it tomorrow.

Sorry, I missed a fix in the experimental branch. I'll push out a rebased version in a few minutes.

Quoted 6 lines
Quoted 1 line

Ok, the rebased version is ready to use.

New results are up for both with and without nodatacow. Not much change.

http://btrfs.boxacle.net/repository/raid/history/History.html

Have another run going with nodatacow and ssd.

Hi Steve,

I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.

The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?

-chris

#8 lore raw
Quoted 18 lines · Chris Mason
Quoted 16 lines · Steven Pratt
Quoted 10 lines · Chris Mason
Quoted 6 lines · Chris Mason
Quoted 2 lines

Sure, will try to get to it tomorrow.

Sorry, I missed a fix in the experimental branch. I'll push out a rebased version in a few minutes.

Ok, the rebased version is ready to use.

New results are up for both with and without nodatacow. Not much change.

http://btrfs.boxacle.net/repository/raid/history/History.html

Quoted 9 lines
Quoted 2 lines

Have another run going with nodatacow and ssd.

Hi Steve,

I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.

Sure, no problem.

Quoted 4 lines

The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?

Yes, thanks for pointing that out, had so many issues I forgot to update the graphs for other file systems. Just pushed new graphs with data for 2.6.30-rc7 for all the other file systems. This was from your "newformat" branch from June 6th.

Steve

Quoted 2 lines

-chris

#9 lore raw
Quoted 8 lines · Steven Pratt
Quoted 6 lines

Hi Steve,

I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.

Sure, no problem.

Quoted 7 lines
Quoted 3 lines

The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?

Yes, thanks for pointing that out, had so many issues I forgot to update the graphs for other file systems. Just pushed new graphs with data for 2.6.30-rc7 for all the other file systems. This was from your "newformat" branch from June 6th.

I've been tuning the 128 thread large file streaming writes, and found some easy optimizations. While I'm fixing up these patches, could you please do a streaming O_DIRECT write test run for me? I think buffered writeback in general has some problems right now on high end arrays.

On my box 2.6.31-rc5 streaming buffered write with xfs only got at 200MB/s (with the 128 thread ffsb workload). Buffered btrfs goes at 175MB/s.

O_DIRECT btrfs runs at 390MB/s, while XFS varies a bit between 330MB/s and 250MB/s.

I'm using a 1MB write blocksize.

-chris

#10 lore raw

HI,

Do you have any benchmarks against non-raid common workloads? Like say a desktop user? It would be great to compare against ext3, ext4, xfs etc.,

Thanks,

Quoted 38 lines · Chris Mason
Quoted 8 lines · Steven Pratt
Quoted 6 lines

Hi Steve,

I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.

Sure, no problem.

Quoted 7 lines
Quoted 3 lines

The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?

Yes, thanks for pointing that out, had so many issues I forgot to update the graphs for other file systems. Just pushed new graphs with data for 2.6.30-rc7 for all the other file systems. This was from your "newformat" branch from June 6th.

I've been tuning the 128 thread large file streaming writes, and found some easy optimizations. While I'm fixing up these patches, could you please do a streaming O_DIRECT write test run for me? I think buffered writeback in general has some problems right now on high end arrays.

On my box 2.6.31-rc5 streaming buffered write with xfs only got at 200MB/s (with the 128 thread ffsb workload). Buffered btrfs goes at 175MB/s.

O_DIRECT btrfs runs at 390MB/s, while XFS varies a bit between 330MB/s and 250MB/s.

I'm using a 1MB write blocksize.

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#11 lore raw
Quoted 6 lines · debian developer

HI,

Do you have any benchmarks against non-raid common workloads? Like say a desktop user? It would be great to compare against ext3, ext4, xfs etc.,

Yes, have had a little trouble with that box recently, but plenty of results based on the 2.6.29 kernels here:

http://btrfs.boxacle.net/repository/single-disk/History/History.html

If you are not familiar with the runs I have been doing, you can find the details of the benchmarking machine and test procedures here: http://btrfs.boxacle.net/

Steve

Quoted 46 lines

Thanks,

Quoted 42 lines · Chris Mason
Quoted 9 lines · Steven Pratt
Quoted 6 lines

Hi Steve,

I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.

Sure, no problem.

Quoted 9 lines
Quoted 4 lines

The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?

Yes, thanks for pointing that out, had so many issues I forgot to update the graphs for other file systems. Just pushed new graphs with data for 2.6.30-rc7 for all the other file systems. This was from your "newformat" branch from June 6th.

I've been tuning the 128 thread large file streaming writes, and found some easy optimizations. While I'm fixing up these patches, could you please do a streaming O_DIRECT write test run for me? I think buffered writeback in general has some problems right now on high end arrays.

On my box 2.6.31-rc5 streaming buffered write with xfs only got at 200MB/s (with the 128 thread ffsb workload). Buffered btrfs goes at 175MB/s.

O_DIRECT btrfs runs at 390MB/s, while XFS varies a bit between 330MB/s and 250MB/s.

I'm using a 1MB write blocksize.

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#12 lore raw
In reply to Chris Mason
Quoted 26 lines · Chris Mason
Quoted 9 lines · Steven Pratt
Quoted 6 lines

Hi Steve,

I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.

Sure, no problem.

Quoted 9 lines
Quoted 4 lines

The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?

Yes, thanks for pointing that out, had so many issues I forgot to update the graphs for other file systems. Just pushed new graphs with data for 2.6.30-rc7 for all the other file systems. This was from your "newformat" branch from June 6th.

I've been tuning the 128 thread large file streaming writes, and found some easy optimizations. While I'm fixing up these patches, could you please do a streaming O_DIRECT write test run for me? I think buffered writeback in general has some problems right now on high end arrays.

Quoted 9 lines

On my box 2.6.31-rc5 streaming buffered write with xfs only got at 200MB/s (with the 128 thread ffsb workload). Buffered btrfs goes at 175MB/s.

O_DIRECT btrfs runs at 390MB/s, while XFS varies a bit between 330MB/s and 250MB/s.

I'm using a 1MB write blocksize.

On my todo list, but am swamped this week trying to get ready for vacation. Will try to get to it as soon as I can.

Stee

Quoted 2 lines

-chris

#13 lore raw
Quoted 26 lines · Steven Pratt
Quoted 25 lines · Chris Mason
Quoted 7 lines · Steven Pratt
Quoted 5 lines

Hi Steve,

I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.

Sure, no problem.

Quoted 7 lines
Quoted 3 lines

The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?

Yes, thanks for pointing that out, had so many issues I forgot to update the graphs for other file systems. Just pushed new graphs with data for 2.6.30-rc7 for all the other file systems. This was from your "newformat" branch from June 6th.

I've been tuning the 128 thread large file streaming writes, and found some easy optimizations. While I'm fixing up these patches, could you please do a streaming O_DIRECT write test run for me? I think buffered writeback in general has some problems right now on high end arrays.

On my box 2.6.31-rc5 streaming buffered write with xfs only got at 200MB/s (with the 128 thread ffsb workload). Buffered btrfs goes at 175MB/s.

Quoted 6 lines
Quoted 4 lines

O_DIRECT btrfs runs at 390MB/s, while XFS varies a bit between 330MB/s and 250MB/s.

I'm using a 1MB write blocksize.

On my todo list, but am swamped this week trying to get ready for vacation. Will try to get to it as soon as I can.

Ok, I've pushed out a very raw version of my buffered write fixes to a new branch named performance on btrfs-unstable.

Please try this with the streaming large file create workload. I'm also curious to see if it improves on your box when you mount with

mount -o thread_pool=128

-chris

#14 lore raw
Quoted 43 lines · Chris Mason
Quoted 32 lines · Steven Pratt
Quoted 30 lines · Chris Mason
Quoted 13 lines · Steven Pratt
Quoted 6 lines

Hi Steve,

I think I'm going to start tuning something other than the random-writes, there is definitely low hanging fruit in the large file creates workload ;) Thanks again for posting all of these.

Sure, no problem.

Quoted 4 lines

The history graph has 2.6.31-rc btrfs against 2.6.29-rc ext4. Have you done more recent runs on ext4?

Quoted 5 lines

Yes, thanks for pointing that out, had so many issues I forgot to update the graphs for other file systems. Just pushed new graphs with data for 2.6.30-rc7 for all the other file systems. This was

from your "newformat" branch from June 6th.

I've been tuning the 128 thread large file streaming writes, and found some easy optimizations. While I'm fixing up these patches, could you please do a streaming O_DIRECT write test run for me? I think buffered writeback in general has some problems right now on high end arrays.

On my box 2.6.31-rc5 streaming buffered write with xfs only got at 200MB/s (with the 128 thread ffsb workload). Buffered btrfs goes at 175MB/s.

Quoted 8 lines
Quoted 5 lines

O_DIRECT btrfs runs at 390MB/s, while XFS varies a bit between 330MB/s and 250MB/s.

I'm using a 1MB write blocksize.

On my todo list, but am swamped this week trying to get ready for vacation. Will try to get to it as soon as I can.

Quoted 9 lines

Ok, I've pushed out a very raw version of my buffered write fixes to a new branch named performance on btrfs-unstable.

Please try this with the streaming large file create workload. I'm also curious to see if it improves on your box when you mount with

mount -o thread_pool=128

Better late than never. Finally got this finished up. Mixed bag on this one. BTRFS lags significantly on single threaded. Seems unable to keep IO outstanding to the device. Less that 60% busy on the DM device, compared to 97%+ for all other filesystems. nodatacow helps out, increasing utilization to about 70%, but still trails by a large margin.

Results are more favorable for multithreaded tests. nodatacow is actually the top performer here! However, cow still raises it's ugly head and causes significant performance degradation (45%) and increased CPU (43%). Also, even without cow, BTRFS is consuming 8-10x more CPU than other File Systems. I don't have oprofile data for these runs, as that was causing some issues with BTRFS. Will retry, and see if that problem is fixed.

thread_pool seemed to make no difference at all.

All runs were done against an August 20th pull of experimental tree. These are 1M, odirect file creates, with each file being 1G in size.

Results can be found here:

http://btrfs.boxacle.net/repository/raid/large_create_test/write-test/1M_odirect_create.html

Steve

Quoted 2 lines

-chris

#15 lore raw
Quoted 6 lines · Steven Pratt

Better late than never. Finally got this finished up. Mixed bag on this one. BTRFS lags significantly on single threaded. Seems unable to keep IO outstanding to the device. Less that 60% busy on the DM device, compared to 97%+ for all other filesystems. nodatacow helps out, increasing utilization to about 70%, but still trails by a large margin.

Hi Steve,

Jens Axboe did some profiling on his big test rig and I think we found the biggest CPU problems. The end result is now setting in the master branch of the btrfs-unstable repo.

On his boxes, btrfs went from around 400MB/s streaming writes to 1GB/s limit, and we're now tied with XFS while using less CPU time.

Hopefully you will see similar results ;)

-chris

#16 lore raw
Quoted 12 lines · Chris Mason
Quoted 7 lines · Steven Pratt

Better late than never. Finally got this finished up. Mixed bag on this one. BTRFS lags significantly on single threaded. Seems unable to keep IO outstanding to the device. Less that 60% busy on the DM device, compared to 97%+ for all other filesystems. nodatacow helps out, increasing utilization to about 70%, but still trails by a large margin.

Hi Steve,

Quoted 9 lines

Jens Axboe did some profiling on his big test rig and I think we found the biggest CPU problems. The end result is now setting in the master branch of the btrfs-unstable repo.

On his boxes, btrfs went from around 400MB/s streaming writes to 1GB/s limit, and we're now tied with XFS while using less CPU time.

Hopefully you will see similar results ;)

Hmmm, well no I didn't. Throughputs at 1 and 128 threads are pretty much unchanged, although I do see a good CPU savings on the 128 thread case (with cow). For 16 threads we actually regressed with cow enabled.

Results are here:

http://btrfs.boxacle.net/repository/raid/large_create_test/write-test/1M_odirect_create.html

I'll try to look more into this next week.

Steve

Quoted 6 lines

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#17 lore raw
Quoted 24 lines · Steven Pratt
Quoted 10 lines · Chris Mason
Quoted 6 lines · Steven Pratt

Better late than never. Finally got this finished up. Mixed bag on this one. BTRFS lags significantly on single threaded. Seems unable to keep IO outstanding to the device. Less that 60% busy on the DM device, compared to 97%+ for all other filesystems. nodatacow helps out, increasing utilization to about 70%, but still trails by a large margin.

Hi Steve,

Quoted 8 lines

Jens Axboe did some profiling on his big test rig and I think we found the biggest CPU problems. The end result is now setting in the master branch of the btrfs-unstable repo.

On his boxes, btrfs went from around 400MB/s streaming writes to 1GB/s limit, and we're now tied with XFS while using less CPU time.

Hopefully you will see similar results ;)

Hmmm, well no I didn't. Throughputs at 1 and 128 threads are pretty much unchanged, although I do see a good CPU savings on the 128 thread case (with cow). For 16 threads we actually regressed with cow enabled.

Quoted 6 lines

Results are here:

http://btrfs.boxacle.net/repository/raid/large_create_test/write-test/1M_odirect_create.html

I'll try to look more into this next week.

Hmmm, Jens was benchmarking buffered writes, but he was also testing on his new per-bdi write back code. If your next run could be buffered instead of O_DIRECT, I'd be curious to see the results.

Thanks,
Chris

#18 lore raw
Quoted 27 lines · Chris Mason
Quoted 26 lines · Steven Pratt
Quoted 10 lines · Chris Mason
Quoted 6 lines · Steven Pratt

Better late than never. Finally got this finished up. Mixed bag on this one. BTRFS lags significantly on single threaded. Seems unable to keep IO outstanding to the device. Less that 60% busy on the DM device, compared to 97%+ for all other filesystems. nodatacow helps out, increasing utilization to about 70%, but still trails by a large margin.

Hi Steve,

Quoted 8 lines

Jens Axboe did some profiling on his big test rig and I think we found the biggest CPU problems. The end result is now setting in the master branch of the btrfs-unstable repo.

On his boxes, btrfs went from around 400MB/s streaming writes to 1GB/s limit, and we're now tied with XFS while using less CPU time.

Hopefully you will see similar results ;)

Hmmm, well no I didn't. Throughputs at 1 and 128 threads are pretty much unchanged, although I do see a good CPU savings on the 128 thread case (with cow). For 16 threads we actually regressed with cow enabled.

Results are here:

Quoted 8 lines
Quoted 4 lines

http://btrfs.boxacle.net/repository/raid/large_create_test/write-test/1M_odirect_create.html

I'll try to look more into this next week.

Hmmm, Jens was benchmarking buffered writes, but he was also testing on his new per-bdi write back code. If your next run could be buffered instead of O_DIRECT, I'd be curious to see the results.

I found out today that a larger MAX_WRITEBACK_PAGES is still an essential for me. It basically doubles throughput on btrfs. So I think we need to do something about that, and sooner and rather than later.

Signature · 1 line
Jens Axboe
#19 lore raw
In reply to Chris Mason
Quoted 33 lines · Chris Mason
Quoted 31 lines · Steven Pratt
Quoted 11 lines · Chris Mason
Quoted 7 lines · Steven Pratt

Better late than never. Finally got this finished up. Mixed bag on this one. BTRFS lags significantly on single threaded. Seems unable to keep IO outstanding to the device. Less that 60% busy on the DM device, compared to 97%+ for all other filesystems. nodatacow helps out, increasing utilization to about 70%, but still trails by a large margin.

Hi Steve,

Quoted 9 lines

Jens Axboe did some profiling on his big test rig and I think we found the biggest CPU problems. The end result is now setting in the master branch of the btrfs-unstable repo.

On his boxes, btrfs went from around 400MB/s streaming writes to 1GB/s limit, and we're now tied with XFS while using less CPU time.

Hopefully you will see similar results ;)

Hmmm, well no I didn't. Throughputs at 1 and 128 threads are pretty much unchanged, although I do see a good CPU savings on the 128 thread case (with cow). For 16 threads we actually regressed with cow enabled.

Results are here:

http://btrfs.boxacle.net/repository/raid/large_create_test/write-test/1M_odirect_create.html

Quoted 9 lines
Quoted 3 lines

I'll try to look more into this next week.

Hmmm, Jens was benchmarking buffered writes, but he was also testing on his new per-bdi write back code. If your next run could be buffered instead of O_DIRECT, I'd be curious to see the results.

Buffered does look a lot better. I don't have a btrfs baseline before these latest changes for this exact workload, but these results are not bad at all. With cow, beats just about everything except XFS, and with nocow simply screams. CPU consumption looks good as well. I'll probably give the full set of tests a run tonight.

Results are here:
http://btrfs.boxacle.net/repository/raid/buffered-creates/buffered-create/buffered-create.html

Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:

btrfs1 kernel: [251789.525886] ------------[ cut here ]------------
Message from syslogd@ at Mon Sep 14 13:13:04 2009 ...
btrfs1 kernel: [251789.526574] invalid opcode: 0000 [#1] SMP
Message from syslogd@ at Mon Sep 14 13:13:04 2009 ...
btrfs1 kernel: [251789.526654] last sysfs file: /sys/devices/pci0000:0c/0000:0c:01.0/local_cpus
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] Stack:
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] ffff88013fc234c0 ffff88013fbcf400 0000000000000000 ffff88013fc01080
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] ffff880132e11d38 ffffffff802a5392 0000000000000001 ffff88013fbcf400
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] Call Trace:
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802a5392>] cache_flusharray+0x7d/0xae
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802a5629>] kfree+0x192/0x1b1
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffffa0378c9f>] put_worker+0x14/0x16 [btrfs]
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffffa0378d55>] btrfs_stop_workers+0xb4/0xc9 [btrfs]
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffffa0355cbe>] close_ctree+0x210/0x288 [btrfs]
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802bd1a1>] ? invalidate_inodes+0x100/0x112
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffffa033f4cb>] btrfs_put_super+0x18/0x27 [btrfs]
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802ad12b>] generic_shutdown_super+0x73/0xe2
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802ad1e5>] kill_anon_super+0x11/0x3b
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802ad51d>] deactivate_super+0x62/0x77
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802bf9eb>] mntput_no_expire+0xec/0x12c
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff802bff3a>] sys_umount+0x2c5/0x31c
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] [<ffffffff8020ba2b>] system_call_fastpath+0x16/0x1b
Message from syslogd@ at Mon Sep 14 13:13:05 2009 ...
btrfs1 kernel: [251789.526654] Code: 89 f7 e8 48 07 f8 ff 48 c1 e8 0c 48 ba 00 00 00 00 00 e2 ff ff 48 6b c0 38 48 01 d0 66 83 38 00 79 04 48 8b 40 10 80 38 00 78 04 <0f> 0b eb fe 48 8b 58 30 48 63 45 c8 48 89 df 4d 8b a4 c5 60 08

Steve

Quoted 3 lines

Thanks,
Chris

#20 lore raw
Quoted 36 lines · Steven Pratt
Quoted 26 lines · Chris Mason
Quoted 25 lines · Steven Pratt
Quoted 9 lines · Chris Mason
Quoted 6 lines · Steven Pratt

Better late than never. Finally got this finished up. Mixed bag on this one. BTRFS lags significantly on single threaded. Seems unable to keep IO outstanding to the device. Less that 60% busy on the DM device, compared to 97%+ for all other filesystems. nodatacow helps out, increasing utilization to about 70%, but still trails by a large margin.

Hi Steve,

Quoted 8 lines

Jens Axboe did some profiling on his big test rig and I think we found the biggest CPU problems. The end result is now setting in the master branch of the btrfs-unstable repo.

On his boxes, btrfs went from around 400MB/s streaming writes to 1GB/s limit, and we're now tied with XFS while using less CPU time.

Hopefully you will see similar results ;)

Hmmm, well no I didn't. Throughputs at 1 and 128 threads are pretty much unchanged, although I do see a good CPU savings on the 128 thread case (with cow). For 16 threads we actually regressed with cow enabled.

Results are here:

Quoted 9 lines
Quoted 4 lines

http://btrfs.boxacle.net/repository/raid/large_create_test/write-test/1M_odirect_create.html

I'll try to look more into this next week.

Hmmm, Jens was benchmarking buffered writes, but he was also testing on his new per-bdi write back code. If your next run could be buffered instead of O_DIRECT, I'd be curious to see the results.

Quoted 6 lines

Buffered does look a lot better. I don't have a btrfs baseline before these latest changes for this exact workload, but these results are not bad at all. With cow, beats just about everything except XFS, and with nocow simply screams. CPU consumption looks good as well. I'll probably give the full set of tests a run tonight.

Wow, good news at last ;) For the oops, try the patch below (I need to push it out, but I think it'll help).

I'll try to figure out the O_DIRECT problems.

-chris

fs/btrfs/async-thread.c +2 -2
index 6ea5cd0..ba28742 100644
@@ -177,7 +177,7 @@ static int try_worker_shutdown(struct btrfs_worker_thread *worker)
 	int freeit = 0;
 
 	spin_lock_irq(&worker->lock);
-	spin_lock_irq(&worker->workers->lock);
+	spin_lock(&worker->workers->lock);
 	if (worker->workers->num_workers > 1 &&
 	    worker->idle &&
 	    !worker->working &&
@@ -188,7 +188,7 @@ static int try_worker_shutdown(struct btrfs_worker_thread *worker)
 		list_del_init(&worker->worker_list);
 		worker->workers->num_workers--;
 	}
-	spin_unlock_irq(&worker->workers->lock);
+	spin_unlock(&worker->workers->lock);
 	spin_unlock_irq(&worker->lock);
 
 	if (freeit)
#21 lore raw
In reply to Steven Pratt
Quoted 4 lines · Steven Pratt

Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:

I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.

-chris

#22 lore raw
Quoted 11 lines · Chris Mason
Quoted 4 lines · Steven Pratt

Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:

I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.

Glad you said that. Keeps me from sending the email that said the patch didn't help :-)

Steve

Quoted 7 lines

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#23 lore raw
Quoted 8 lines · Steven Pratt
Quoted 7 lines · Chris Mason
Quoted 4 lines · Steven Pratt

Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:

Quoted 7 lines
Quoted 4 lines

I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.

Glad you said that. Keeps me from sending the email that said the patch didn't help :-)

Steve

Well, still getting oopses even with new code.

Lots of:

Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] BUG: soft lockup - CPU#10 stuck for 61s! [btrfs-endio-1:30250]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Pid: 30250, comm: btrfs-endio-1 Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]-
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RIP: 0010:[<ffffffff81153920>]  [<ffffffff81153920>] crc32c+0x20/0x26
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RSP: 0018:ffff88013a857cc8  EFLAGS: 00000217
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RAX: 0000000000000040 RBX: ffff88013a857cc8 RCX: ffff88013d8022c0
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RDX: 0000000000000010 RSI: ffff88001d349ff0 RDI: 0000000041703e71
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RBP: ffffffff8100c4ee R08: 0000000000000000 R09: 0000000000000000
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] R10: ffff88013a857d30 R11: 0000000000000002 R12: ffff88013a857d10
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] R13: 0000000000000002 R14: ffff88013a857cb0 R15: ffffffff8100c38e
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] FS:  0000000000000000(0000) GS:ffff880028159000(0000) knlGS:0000000000000000
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] CR2: 0000000000000043 CR3: 00000001368f7000 CR4: 00000000000006e0
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Call Trace:
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754]  [<ffffffff8115397e>] ? chksum_update+0x10/0x18
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754]  [<ffffffff81150084>] ? crypto_shash_update+0x1a/0x1c
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754]  [<ffffffff81175c34>] ? crc32c+0x4c/0x60
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754]  [<ffffffffa0391d0f>] ? get_state_private+0x38/0x6f [btrfs]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754]  [<ffffffffa0376688>] ? btrfs_csum_data+0xd/0xf [btrfs]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754]  [<ffffffffa037fefc>] ? btrfs_readpage_end_io_hook+0x158/0x27b [btrfs]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754]  [<ffffffffa0392a46>] ? end_bio_extent_readpage+0xb8/0x1c0 [btrfs]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754]  [<ffffffff810e5733>] ? bio_endio+0x26/0x28
Sep 16 11:07:27 btrfs1 kernel: [ 1862.947656]  [<ffffffffa037666e>] ? end_workqueue_fn+0x111/0x11e [btrfs]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.947823]  [<ffffffffa039a490>] ? worker_loop+0x12a/0x3ea [btrfs]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.947823]  [<ffffffffa039a366>] ? worker_loop+0x0/0x3ea [btrfs]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.948800]  [<ffffffff810544e4>] ? kthread+0x8f/0x97
Sep 16 11:07:27 btrfs1 kernel: [ 1862.948800]  [<ffffffff8100ca1a>] ? child_rip+0xa/0x20
Sep 16 11:07:27 btrfs1 kernel: [ 1862.948800]  [<ffffffff81054455>] ? kthread+0x0/0x97
Sep 16 11:07:27 btrfs1 kernel: [ 1862.948800]  [<ffffffff8100ca10>] ? child_rip+0x0/0x20
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] Pid: 31421, comm: btrfs-endio-wri Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]-
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] RIP: 0010:[<ffffffffa036afb3>]  [<ffffffffa036afb3>] alloc_reserved_file_extent+0x8d/0x1c3 [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] RSP: 0018:ffff8800aa555af0  EFLAGS: 00010282
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] RAX: 00000000ffffffef RBX: ffff88013b55e000 RCX: 0000000000000002
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff88012f20a9a0
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] RBP: ffff8800aa555b60 R08: ffff8800aa555888 R09: ffff8800aa555880
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] R10: ffff880077937400 R11: 00000000fffffffa R12: 000000000000001d
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] R13: ffff880077937400 R14: 0000000000000000 R15: 0000000000000000
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] FS:  0000000000000000(0000) GS:ffff88002804b000(0000) knlGS:0000000000000000
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] CR2: 00000000007c0000 CR3: 000000013e038000 CR4: 00000000000006f0
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] Process btrfs-endio-wri (pid: 31421, threadinfo ffff8800aa554000, task ffff8801395447a0)
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] Stack:
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  ffff880077937400 0000000000000a7c 0000000000000005 0000000000000000
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] <0> ffff880101d0c800 ffff8801140bbd20 000000b2aa555b60 ffffffffa036a190
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] <0> 000000350000091d ffff8801090fdd40 ffff88013a4e9d40 0000000000000001
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] Call Trace:
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  [<ffffffffa036a190>] ? update_reserved_extents+0xa7/0xbe [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  [<ffffffffa036f430>] run_one_delayed_ref+0x382/0x42f [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  [<ffffffff8100c4ee>] ? apic_timer_interrupt+0xe/0x20
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  [<ffffffffa03700b1>] run_clustered_refs+0x237/0x2b4 [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  [<ffffffffa03a5665>] ? btrfs_find_ref_cluster+0xdc/0x115 [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  [<ffffffffa03701da>] btrfs_run_delayed_refs+0xac/0x195 [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  [<ffffffffa0379a76>] __btrfs_end_transaction+0x59/0xfe [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  [<ffffffffa0379b36>] btrfs_end_transaction+0xb/0xd [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  [<ffffffffa037f29b>] btrfs_finish_ordered_io+0x23c/0x265 [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  [<ffffffffa037f2d9>] btrfs_writepage_end_io_hook+0x15/0x17 [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  [<ffffffffa0392901>] end_bio_extent_writepage+0xa5/0x132 [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  [<ffffffff810e5733>] bio_endio+0x26/0x28
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  [<ffffffffa037666e>] end_workqueue_fn+0x111/0x11e [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  [<ffffffffa039a490>] worker_loop+0x12a/0x3ea [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621]  [<ffffffffa039a366>] ? worker_loop+0x0/0x3ea [btrfs]
Sep 16 11:54:48 btrfs1 kernel: [ 4703.082621]  [<ffffffff810544e4>] kthread+0x8f/0x97
Sep 16 11:54:48 btrfs1 kernel: [ 4703.082621]  [<ffffffff8100ca1a>] child_rip+0xa/0x20
Sep 16 11:54:48 btrfs1 kernel: [ 4703.082621]  [<ffffffff81054455>] ? kthread+0x0/0x97
Sep 16 11:54:48 btrfs1 kernel: [ 4703.082621]  [<ffffffff8100ca10>] ? child_rip+0x0/0x20
Sep 16 11:54:48 btrfs1 kernel: [ 4703.082621] Code: 08 4c 8d 45 d4 41 8d 44 24 18 48 8b 73 20 48 8b 4d 18 41 b9 01 00 00 00 48 8b 7d b8 4c 89 ea

89 45 d4 e8 93 e3 ff ff 85 c0 74 04 <0f> 0b eb fe 49 63 75 40 4d 8b 65 00 49 83 cf 01 4c 89 e7 48 6b

Happened on 2 machines.

Steve

Quoted 13 lines
Quoted 8 lines

-chris

Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#24 lore raw
Quoted 16 lines · Steven Pratt
Quoted 6 lines · Steven Pratt
Quoted 5 lines · Chris Mason
Quoted 3 lines · Steven Pratt

Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:

Quoted 7 lines
Quoted 3 lines

I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.

Glad you said that. Keeps me from sending the email that said the patch didn't help :-)

Steve

Well, still getting oopses even with new code.

Quoted 7 lines

Lots of:

Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] BUG: soft lockup -
CPU#10 stuck for 61s! [btrfs-endio-1:30250]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Pid: 30250, comm:
btrfs-endio-1 Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]-
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RIP:

0010:[<ffffffff81153920>] [<ffffffff81153920>] crc32c+0x20/0x26

If I'm reading this right, you've got a softlockup in crc32c? Something has gone really wrong here. Are you reusing datasets from old runs?

-chris

#25 lore raw
Quoted 32 lines · Chris Mason
Quoted 21 lines · Steven Pratt
Quoted 8 lines · Steven Pratt
Quoted 6 lines · Chris Mason
Quoted 4 lines · Steven Pratt

Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:

Quoted 9 lines
Quoted 4 lines

I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.

Glad you said that. Keeps me from sending the email that said the patch didn't help :-)

Steve

Well, still getting oopses even with new code.

Quoted 8 lines

Lots of:

Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] BUG: soft lockup -
CPU#10 stuck for 61s! [btrfs-endio-1:30250]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Pid: 30250, comm:
btrfs-endio-1 Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]-
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RIP:

0010:[<ffffffff81153920>] [<ffffffff81153920>] crc32c+0x20/0x26

Quoted 3 lines

If I'm reading this right, you've got a softlockup in crc32c? Something has gone really wrong here. Are you reusing datasets from old runs?

No, mkfs before every run.

Steve

Quoted 6 lines

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#26 lore raw
Quoted 25 lines · Steven Pratt
Quoted 24 lines · Chris Mason
Quoted 22 lines · Steven Pratt
Quoted 5 lines · Steven Pratt
Quoted 4 lines · Chris Mason
Quoted 3 lines · Steven Pratt

Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:

Quoted 7 lines
Quoted 3 lines

I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.

Glad you said that. Keeps me from sending the email that said the patch didn't help :-)

Steve

Well, still getting oopses even with new code.

Lots of:

Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] BUG: soft lockup -
CPU#10 stuck for 61s! [btrfs-endio-1:30250]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Pid: 30250, comm:
btrfs-endio-1 Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]-
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RIP:

0010:[<ffffffff81153920>] [<ffffffff81153920>] crc32c+0x20/0x26

Quoted 3 lines
Quoted 2 lines

If I'm reading this right, you've got a softlockup in crc32c? Something has gone really wrong here. Are you reusing datasets from old runs?

No, mkfs before every run.

Could you please send me the full softlockup output? Its hard to read all line wrapped, so the original files would help.

-chris

#27 lore raw
In reply to Chris Mason
Quoted 32 lines · Chris Mason
Quoted 21 lines · Steven Pratt
Quoted 8 lines · Steven Pratt
Quoted 6 lines · Chris Mason
Quoted 4 lines · Steven Pratt

Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:

Quoted 9 lines
Quoted 4 lines

I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.

Glad you said that. Keeps me from sending the email that said the patch didn't help :-)

Steve

Well, still getting oopses even with new code.

Quoted 8 lines

Lots of:

Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] BUG: soft lockup -
CPU#10 stuck for 61s! [btrfs-endio-1:30250]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Pid: 30250, comm:
btrfs-endio-1 Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]-
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RIP:

0010:[<ffffffff81153920>] [<ffffffff81153920>] crc32c+0x20/0x26

Quoted 3 lines

If I'm reading this right, you've got a softlockup in crc32c? Something has gone really wrong here. Are you reusing datasets from old runs?

From the second machine a single bug:

Sep 16 11:53:42 btrfs2 kernel: [ 3769.298240] ------------[ cut here ]------------
Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] kernel BUG at fs/btrfs/extent-tree.c:4097!
Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] invalid opcode: 0000 [#1] SMP
Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] last sysfs file: /sys/devices/system/cpu/cpu15/cache/index1/shared_cpu_map
Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] CPU 9
Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler btrfs zlib_deflate oprofile autofs4 nfs lockd nfs_acl auth_rpc
gss sunrpc dm_multipath video output sbs sbshc battery ac parport_pc lp parport sg joydev serio_raw acpi_memhotplug rtc_cmos rtc_core rtc_lib button tg3 libphy i2c_
piix4 i2c_core pcspkr dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod lpfc scsi_transport_fc aic94xx libsas libata scsi_transport_sas sd_mod scsi_mod ext
3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] Pid: 2106, comm: btrfs-endio-wri Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]-
Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] RIP: 0010:[<ffffffffa0386fb3>]  [<ffffffffa0386fb3>] alloc_reserved_file_extent+0x8d/0x1c3 [btrfs]
Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] RSP: 0018:ffff88002758faf0  EFLAGS: 00010282
Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] RAX: 00000000ffffffef RBX: ffff880136434000 RCX: 0000000000000002
Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8800a7040370
Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] RBP: ffff88002758fb60 R08: ffff88002758f958 R09: ffff88002758f950
Sep 16 11:53:42 btrfs2 kernel: [ 3769.298550] R10: 0000000000000004 R11: ffff8800a7040370 R12: 000000000000001d
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] R13: ffff8800b79e6910 R14: 0000000000000000 R15: 0000000000000000
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] FS:  0000000000000000(0000) GS:ffff88002813e000(0000) knlGS:0000000000000000
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] CR2: 00007f1f6915a000 CR3: 000000013dd4e000 CR4: 00000000000006e0
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] Process btrfs-endio-wri (pid: 2106, threadinfo ffff88002758e000, task ffff88013b94c100)
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] Stack:
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  ffff8800709fc760 0000000000000856 0000000000000005 0000000000000000
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] <0> ffff8801329d5000 ffff880102242de0 000000b22758fb60 ffffffffa0386190
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] <0> 00000035329d5000 ffff880128291440 ffff880108302340 0000000000000001
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] Call Trace:
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffffa0386190>] ? update_reserved_extents+0xa7/0xbe [btrfs]
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffffa038b430>] run_one_delayed_ref+0x382/0x42f [btrfs]
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffffa038c0b1>] run_clustered_refs+0x237/0x2b4 [btrfs]
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffffa03c1665>] ? btrfs_find_ref_cluster+0xdc/0x115 [btrfs]
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffffa038c1da>] btrfs_run_delayed_refs+0xac/0x195 [btrfs]
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffffa0395a76>] __btrfs_end_transaction+0x59/0xfe [btrfs]
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffffa0395b36>] btrfs_end_transaction+0xb/0xd [btrfs]
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffffa039b29b>] btrfs_finish_ordered_io+0x23c/0x265 [btrfs]
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffffa039b2d9>] btrfs_writepage_end_io_hook+0x15/0x17 [btrfs]
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffffa03ae901>] end_bio_extent_writepage+0xa5/0x132 [btrfs]
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffff810e5733>] bio_endio+0x26/0x28
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffffa039266e>] end_workqueue_fn+0x111/0x11e [btrfs]
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffffa03b6490>] worker_loop+0x12a/0x3ea [btrfs]
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffffa03b6366>] ? worker_loop+0x0/0x3ea [btrfs]
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffff810544e4>] kthread+0x8f/0x97
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffff8100ca1a>] child_rip+0xa/0x20
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffff81054455>] ? kthread+0x0/0x97
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550]  [<ffffffff8100ca10>] ? child_rip+0x0/0x20
Sep 16 11:53:43 btrfs2 kernel: [ 3769.298550] Code: 08 4c 8d 45 d4 41 8d 44 24 18 48 8b 73 20 48 8b 4d 18 41 b9 01 00 00 00 48 8b 7d b8 4c 89 ea 89 45 d4 e8 93 e3 f

f ff 85 c0 74 04 <0f> 0b eb fe 49 63 75 40 4d 8b 65 00 49 83 cf 01 4c 89 e7 48 6b

Steve

Quoted 6 lines

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#28 lore raw
Quoted 25 lines · Steven Pratt
Quoted 24 lines · Chris Mason
Quoted 22 lines · Steven Pratt
Quoted 5 lines · Steven Pratt
Quoted 4 lines · Chris Mason
Quoted 3 lines · Steven Pratt

Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:

Quoted 7 lines
Quoted 3 lines

I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.

Glad you said that. Keeps me from sending the email that said the patch didn't help :-)

Steve

Well, still getting oopses even with new code.

Lots of:

Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] BUG: soft lockup -
CPU#10 stuck for 61s! [btrfs-endio-1:30250]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Pid: 30250, comm:
btrfs-endio-1 Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]-
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RIP:

0010:[<ffffffff81153920>] [<ffffffff81153920>] crc32c+0x20/0x26

Quoted 4 lines
Quoted 2 lines

If I'm reading this right, you've got a softlockup in crc32c? Something has gone really wrong here. Are you reusing datasets from old runs?

From the second machine a single bug:

Sep 16 11:53:42 btrfs2 kernel: [ 3769.298240] ------------[ cut here

Ok, which mount options and job file is this from?

-chris

#29 lore raw
Quoted 35 lines · Chris Mason
Quoted 33 lines · Steven Pratt
Quoted 31 lines · Chris Mason
Quoted 29 lines · Steven Pratt
Quoted 8 lines · Steven Pratt
Quoted 6 lines · Chris Mason
Quoted 4 lines · Steven Pratt

Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:

Quoted 9 lines
Quoted 4 lines

I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.

Glad you said that. Keeps me from sending the email that said the patch didn't help :-)

Steve

Well, still getting oopses even with new code.

Lots of:

Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] BUG: soft lockup -
CPU#10 stuck for 61s! [btrfs-endio-1:30250]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Pid: 30250, comm:
btrfs-endio-1 Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]-
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RIP:

0010:[<ffffffff81153920>] [<ffffffff81153920>] crc32c+0x20/0x26

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines

If I'm reading this right, you've got a softlockup in crc32c? Something has gone really wrong here. Are you reusing datasets from old runs?

From the second machine a single bug:

Sep 16 11:53:42 btrfs2 kernel: [ 3769.298240] ------------[ cut here

Ok, which mount options and job file is this from?

mount -t btrfs /dev/ffsbdev1 /mnt/ffsb1'
[20090916-11:47:37.738883526] PROCESSING COMMAND : 'run random_writes__threads_0001 ffsb http://hks.austin.ibm.com/users/corry/btrfs/ffsb/profiles/btrfs2/random_writes.ffsb num_threads=1'

So , this is single disk machine, running single threaded random write workload. Buffered, not odirect.

I'm packaging up full messages file, repeated errros makes it big. Will send separately.

Steve

Quoted 6 lines

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#30 lore raw
In reply to Chris Mason
Quoted 8 lines · Chris Mason
Quoted 3 lines · Steven Pratt

Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:

I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.

Chris:

FYI - all five of my test systems have now finished my standard test cycle on the -unstable master branch, and I've not seen a single hang. So, your fix for the async thread shutdown race seems to have fixed my problems, even if Steve's still seeing trouble.

I'll note that the running times for fsstress on some of my systems have become rather longer with btrfs-unstable/master kernels - 3.5 rather than 2.5 hours on multidevice filesystems. Running times on single device filesystems are roughly the same.

I'm going to start another set of tests for thoroughness unless you've got more patches coming.

Thanks,
Eric

Quoted 3 lines

-chris

#31 lore raw
Quoted 12 lines · Eric Whitney
Quoted 8 lines · Chris Mason
Quoted 3 lines · Steven Pratt

Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:

I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.

Quoted 7 lines

Chris:

FYI - all five of my test systems have now finished my standard test cycle on the -unstable master branch, and I've not seen a single hang. So, your fix for the async thread shutdown race seems to have fixed my problems, even if Steve's still seeing trouble.

I'll note that the running times for fsstress on some of my systems have become rather longer with btrfs-unstable/master kernels - 3.5 rather than 2.5 hours on multidevice filesystems. Running times on single device filesystems are roughly the same.

I'm going to start another set of tests for thoroughness unless you've got more patches coming.

I've had some offline discussions with Chris, and it seems the problem is triggered by unmounting and re-mounting the file system between tests (but not running mkfs again). I have also just verified that the problem does not occur if repeated tests are run without the unmount mount cycle. So in case this is not clear:

mkfs
mount
create new files
run test
umount
mount
delete old files
create new files
run test
BUG

but..

mkfs
mount
create new files
run test
umount
mkfs <------ differnet
mount
delete old files
create new files
run test
...
all is fine

or...

mkfs
mount
create new files
run test

# no mounts or mkfs here
delete old files
create new files
run test
...
all is fine

Steve

Quoted 7 lines

Thanks,
Eric

Quoted 3 lines

-chris

#32 lore raw
Quoted 25 lines · Steven Pratt
Quoted 24 lines · Eric Whitney
Quoted 8 lines · Chris Mason
Quoted 3 lines · Steven Pratt

Only bit of bad news is I did get one error that crashed the system on single threaded nocow run. So that data point is missing. Output below:

I hope I've got this fixed. If you pull from the master branch of btrfs-unstable there are fixes for async thread races. The single patch I sent before is included, but not enough.

Chris:

FYI - all five of my test systems have now finished my standard test cycle on the -unstable master branch, and I've not seen a single hang. So, your fix for the async thread shutdown race seems to have fixed my problems, even if Steve's still seeing trouble.

I'll note that the running times for fsstress on some of my systems have become rather longer with btrfs-unstable/master kernels - 3.5 rather than 2.5 hours on multidevice filesystems. Running times on single device filesystems are roughly the same.

Quoted 7 lines
Quoted 2 lines

I'm going to start another set of tests for thoroughness unless you've got more patches coming.

I've had some offline discussions with Chris, and it seems the problem is triggered by unmounting and re-mounting the file system
between tests (but not running mkfs again). I have also just
verified that the problem does not occur if repeated tests are run without the unmount mount cycle. So in case this is not clear:

Ok, I've triggered it here. Next step is trying Yan Zheng's async caching update.

------------[ cut here ]------------
kernel BUG at fs/btrfs/extent-tree.c:4097!

invalid opcode: 0000 [#1] SMP

-chris

#33 lore raw

[ crashes on runs involving unmounts ]

The run is still going here, but it has survived longer than before. I'm trying with Yan Zheng's patch:

From: Yan Zheng <[email protected]>
Date: Fri, 11 Sep 2009 16:11:19 -0400
Subject: [PATCH] Btrfs: improve async block group caching

This patch gets rid of two limitations of async block group caching. The old code delays handling pinned extents when block group is in caching. To allocate logged file extents, the old code need wait until block group is fully cached. To get rid of the limitations, This patch introduces a data structure to track the progress of caching. Base on the caching progress, we know which extents should be added to the free space cache when handling the pinned extents. The logged file extents are also handled in a similar way.

This patch also changes how pinned extents are tracked. The old code uses one tree to track pinned extents, and copy the pinned extents tree at transaction commit time. This patch makes it use two trees to track pinned extents. One tree for extents that are pinned in the running transaction, one tree for extents that can be unpinned. At transaction commit time, we swap the two trees.

---
 fs/btrfs/ctree.h       |   29 ++-
 fs/btrfs/disk-io.c     |    7 +-
 fs/btrfs/extent-tree.c |  586 +++++++++++++++++++++++++++++-------------------
 fs/btrfs/transaction.c |   15 +-
 fs/btrfs/tree-log.c    |    4 +-
 5 files changed, 382 insertions(+), 259 deletions(-)
fs/btrfs/ctree.h +20 -9
index 732d5b8..3b6df71 100644
@@ -726,6 +726,15 @@ enum btrfs_caching_type {
 	BTRFS_CACHE_FINISHED	= 2,
 };
 
+struct btrfs_caching_control {
+	struct list_head list;
+	struct mutex mutex;
+	wait_queue_head_t wait;
+	struct btrfs_block_group_cache *block_group;
+	u64 progress;
+	atomic_t count;
+};
+
 struct btrfs_block_group_cache {
 	struct btrfs_key key;
 	struct btrfs_block_group_item item;
@@ -742,8 +751,9 @@ struct btrfs_block_group_cache {
 	int dirty;
 
 	/* cache tracking stuff */
-	wait_queue_head_t caching_q;
 	int cached;
+	struct btrfs_caching_control *caching_ctl;
+	u64 last_byte_to_unpin;
 
 	struct btrfs_space_info *space_info;
 
@@ -788,7 +798,8 @@ struct btrfs_fs_info {
 	spinlock_t block_group_cache_lock;
 	struct rb_root block_group_cache_tree;
 
-	struct extent_io_tree pinned_extents;
+	struct extent_io_tree freed_extents[2];
+	struct extent_io_tree *pinned_extents;
 
 	/* logical->physical extent mapping */
 	struct btrfs_mapping_tree mapping_tree;
@@ -825,8 +836,6 @@ struct btrfs_fs_info {
 	struct mutex drop_mutex;
 	struct mutex volume_mutex;
 	struct mutex tree_reloc_mutex;
-	struct rw_semaphore extent_commit_sem;
-
 	/*
 	 * this protects the ordered operations list only while we are
 	 * processing all of the entries on it.  This way we make
@@ -835,10 +844,12 @@ struct btrfs_fs_info {
 	 * before jumping into the main commit.
 	 */
 	struct mutex ordered_operations_mutex;
+	struct rw_semaphore extent_commit_sem;
 
 	struct list_head trans_list;
 	struct list_head hashers;
 	struct list_head dead_roots;
+	struct list_head caching_block_groups;
 
 	atomic_t nr_async_submits;
 	atomic_t async_submit_draining;
@@ -1920,8 +1931,8 @@ void btrfs_put_block_group(struct btrfs_block_group_cache *cache);
 int btrfs_run_delayed_refs(struct btrfs_trans_handle *trans,
 			   struct btrfs_root *root, unsigned long count);
 int btrfs_lookup_extent(struct btrfs_root *root, u64 start, u64 len);
-int btrfs_update_pinned_extents(struct btrfs_root *root,
-				u64 bytenr, u64 num, int pin);
+int btrfs_pin_extent(struct btrfs_root *root,
+		     u64 bytenr, u64 num, int reserved);
 int btrfs_drop_leaf_ref(struct btrfs_trans_handle *trans,
 			struct btrfs_root *root, struct extent_buffer *leaf);
 int btrfs_cross_ref_exist(struct btrfs_trans_handle *trans,
@@ -1971,9 +1982,10 @@ int btrfs_free_extent(struct btrfs_trans_handle *trans,
 		      u64 root_objectid, u64 owner, u64 offset);
 
 int btrfs_free_reserved_extent(struct btrfs_root *root, u64 start, u64 len);
+int btrfs_prepare_extent_commit(struct btrfs_trans_handle *trans,
+				struct btrfs_root *root);
 int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans,
-			       struct btrfs_root *root,
-			       struct extent_io_tree *unpin);
+			       struct btrfs_root *root);
 int btrfs_inc_extent_ref(struct btrfs_trans_handle *trans,
 			 struct btrfs_root *root,
 			 u64 bytenr, u64 num_bytes, u64 parent,
@@ -2006,7 +2018,6 @@ void btrfs_delalloc_reserve_space(struct btrfs_root *root, struct inode *inode,
 				 u64 bytes);
 void btrfs_delalloc_free_space(struct btrfs_root *root, struct inode *inode,
 			      u64 bytes);
-void btrfs_free_pinned_extents(struct btrfs_fs_info *info);
 /* ctree.c */
 int btrfs_bin_search(struct extent_buffer *eb, struct btrfs_key *key,
 		     int level, int *slot);
fs/btrfs/disk-io.c +5 -2
index 253da7e..16dae12 100644
@@ -1563,6 +1563,7 @@ struct btrfs_root *open_ctree(struct super_block *sb,
 	INIT_LIST_HEAD(&fs_info->hashers);
 	INIT_LIST_HEAD(&fs_info->delalloc_inodes);
 	INIT_LIST_HEAD(&fs_info->ordered_operations);
+	INIT_LIST_HEAD(&fs_info->caching_block_groups);
 	spin_lock_init(&fs_info->delalloc_lock);
 	spin_lock_init(&fs_info->new_trans_lock);
 	spin_lock_init(&fs_info->ref_cache_lock);
@@ -1621,8 +1622,11 @@ struct btrfs_root *open_ctree(struct super_block *sb,
 	spin_lock_init(&fs_info->block_group_cache_lock);
 	fs_info->block_group_cache_tree.rb_node = NULL;
 
-	extent_io_tree_init(&fs_info->pinned_extents,
+	extent_io_tree_init(&fs_info->freed_extents[0],
 			     fs_info->btree_inode->i_mapping, GFP_NOFS);
+	extent_io_tree_init(&fs_info->freed_extents[1],
+			     fs_info->btree_inode->i_mapping, GFP_NOFS);
+	fs_info->pinned_extents = &fs_info->freed_extents[0];
 	fs_info->do_barriers = 1;
 
 	BTRFS_I(fs_info->btree_inode)->root = tree_root;
@@ -2359,7 +2363,6 @@ int close_ctree(struct btrfs_root *root)
 	free_extent_buffer(root->fs_info->csum_root->commit_root);
 
 	btrfs_free_block_groups(root->fs_info);
-	btrfs_free_pinned_extents(root->fs_info);
 
 	del_fs_roots(fs_info);
 
fs/btrfs/extent-tree.c +352 -234

941 lines not on this page · raw message

fs/btrfs/transaction.c +3 -12
index cdbb502..6ed6186 100644
@@ -874,7 +874,6 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 	unsigned long timeout = 1;
 	struct btrfs_transaction *cur_trans;
 	struct btrfs_transaction *prev_trans = NULL;
-	struct extent_io_tree *pinned_copy;
 	DEFINE_WAIT(wait);
 	int ret;
 	int should_grow = 0;
@@ -915,13 +914,6 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 		return 0;
 	}
 
-	pinned_copy = kmalloc(sizeof(*pinned_copy), GFP_NOFS);
-	if (!pinned_copy)
-		return -ENOMEM;
-
-	extent_io_tree_init(pinned_copy,
-			     root->fs_info->btree_inode->i_mapping, GFP_NOFS);
-
 	trans->transaction->in_commit = 1;
 	trans->transaction->blocked = 1;
 	if (cur_trans->list.prev != &root->fs_info->trans_list) {
@@ -1019,6 +1011,8 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 	ret = commit_cowonly_roots(trans, root);
 	BUG_ON(ret);
 
+	btrfs_prepare_extent_commit(trans, root);
+
 	cur_trans = root->fs_info->running_transaction;
 	spin_lock(&root->fs_info->new_trans_lock);
 	root->fs_info->running_transaction = NULL;
@@ -1042,8 +1036,6 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 	memcpy(&root->fs_info->super_for_commit, &root->fs_info->super_copy,
 	       sizeof(root->fs_info->super_copy));
 
-	btrfs_copy_pinned(root, pinned_copy);
-
 	trans->transaction->blocked = 0;
 
 	wake_up(&root->fs_info->transaction_wait);
@@ -1059,8 +1051,7 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
 	 */
 	mutex_unlock(&root->fs_info->tree_log_mutex);
 
-	btrfs_finish_extent_commit(trans, root, pinned_copy);
-	kfree(pinned_copy);
+	btrfs_finish_extent_commit(trans, root);
 
 	/* do the directory inserts of any pending snapshot creations */
 	finish_pending_snapshots(trans, root->fs_info);
fs/btrfs/tree-log.c +2 -2
index 8661a73..f4a7b62 100644
@@ -263,8 +263,8 @@ static int process_one_buffer(struct btrfs_root *log,
 			      struct walk_control *wc, u64 gen)
 {
 	if (wc->pin)
-		btrfs_update_pinned_extents(log->fs_info->extent_root,
-					    eb->start, eb->len, 1);
+		btrfs_pin_extent(log->fs_info->extent_root,
+				 eb->start, eb->len, 0);
 
 	if (btrfs_buffer_uptodate(eb, gen)) {
 		if (wc->write)
Signature · 1 line
1.6.4.1
#34 lore raw
Quoted 8 lines · Chris Mason

[ crashes on runs involving unmounts ]

The run is still going here, but it has survived longer than before. I'm trying with Yan Zheng's patch:

From: Yan Zheng <[email protected]>
Date: Fri, 11 Sep 2009 16:11:19 -0400
Subject: [PATCH] Btrfs: improve async block group caching

Quick update, I got through a full run of Steve's test with this applied. I'll start a few more ;)

-chris

#35 lore raw
Quoted 7 lines · Chris Mason
Quoted 5 lines · Chris Mason

[ crashes on runs involving unmounts ]

The run is still going here, but it has survived longer than before. I'm trying with Yan Zheng's patch:

Quoted 8 lines
Quoted 4 lines

From: Yan Zheng <[email protected]>
Date: Fri, 11 Sep 2009 16:11:19 -0400
Subject: [PATCH] Btrfs: improve async block group caching

Quick update, I got through a full run of Steve's test with this applied. I'll start a few more ;)

Seems to work for me too! Got through all the random write tests with no problems. Will kick off full run overnight.

Steve

Quoted 6 lines

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#36 lore raw
Quoted 7 lines · Steven Pratt
Quoted 6 lines · Chris Mason
Quoted 5 lines · Chris Mason

[ crashes on runs involving unmounts ]

The run is still going here, but it has survived longer than before. I'm trying with Yan Zheng's patch:

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

From: Yan Zheng <[email protected]>
Date: Fri, 11 Sep 2009 16:11:19 -0400
Subject: [PATCH] Btrfs: improve async block group caching

Quick update, I got through a full run of Steve's test with this applied. I'll start a few more ;)

Seems to work for me too! Got through all the random write tests with no problems. Will kick off full run overnight.

Thanks again.

I've updated the btrfs-unstable tree to include Yan Zheng's fix. I've also included two more buffered writeback speedups, and I expect these to make a difference on the multi-threaded tests.

I took a look at 1MB O_DIRECT writes, and the latencies of sending off checksumming to the checksum threads seem to be the biggest problem. I get full tput at 8MB O_DIRECT writes, so for now I'm going to leave this one alone.

-chris

#37 lore raw
Quoted 24 lines · Chris Mason
Quoted 13 lines · Steven Pratt
Quoted 11 lines · Chris Mason
Quoted 9 lines · Chris Mason

[ crashes on runs involving unmounts ]

The run is still going here, but it has survived longer than before. I'm trying with Yan Zheng's patch:

From: Yan Zheng <[email protected]>
Date: Fri, 11 Sep 2009 16:11:19 -0400
Subject: [PATCH] Btrfs: improve async block group caching

Quoted 6 lines
Quoted 3 lines

Quick update, I got through a full run of Steve's test with this applied. I'll start a few more ;)

Seems to work for me too! Got through all the random write tests with no problems. Will kick off full run overnight.

Thanks again.

Quoted 9 lines

I've updated the btrfs-unstable tree to include Yan Zheng's fix. I've also included two more buffered writeback speedups, and I expect these to make a difference on the multi-threaded tests.

I took a look at 1MB O_DIRECT writes, and the latencies of sending off checksumming to the checksum threads seem to be the biggest problem. I get full tput at 8MB O_DIRECT writes, so for now I'm going to leave this one alone.

Updated performance results are available. Ran both the 9/16 tree + async patches and the tree from 9/18. Results are a mixed bag, some faster some slower.

http://btrfs.boxacle.net/repository/raid/history/History.html

Steve

Quoted 7 lines

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html