Skip to contentSkip to conversation

More random write performance data

DISCUSSION Steven Pratt 2 participants 4 messages Apr 8, 2009 – Apr 9, 2009 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Given the anomalies we were seeing on random write workloads, I decided to simplify the test and do single threaded odirect random write. This should eliminate the locking issue as well as any pdflush bursty behavior. What I got was not quite what I expected.

The most interesting graph is probably #12, DM write throughput. We see a baseline of ~7MB/sec with spikes every 30 seconds. I assume the spike are meta data related as the io is being done from user space at a steady constant rate. The really odd thing is that for the entire almost 2 hour duration, the amplitude of the spike continues to climb, meaning the amount of meta data need to be flushed to disk is ever increasing.

http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun-1thread/btrfs1.ffsb.random_writes__threads_0001.09-04-08_13.05.54/analysis/iostat-processed.001/chart.html

Looking at graph #8 DM IO/sec, we see that there is even a pattern within the pattern of spikes. It # of IOs in each spike appears to change at each interval and repeats over a set of 7, 30 second intervals.

Also, we see that we average 12MB/sec of data written out, for 5MB/sec of benchmark throughput.

I have queued up a run without checksums and cow to see how much this overhead is reduced.

Steve

#1 lore raw
Quoted 19 lines · Steven Pratt

Given the anomalies we were seeing on random write workloads, I decided to simplify the test and do single threaded odirect random write. This should eliminate the locking issue as well as any pdflush bursty behavior. What I got was not quite what I expected.

The most interesting graph is probably #12, DM write throughput. We see a baseline of ~7MB/sec with spikes every 30 seconds. I assume the spike are meta data related as the io is being done from user space at a steady constant rate. The really odd thing is that for the entire almost 2 hour duration, the amplitude of the spike continues to climb, meaning the amount of meta data need to be flushed to disk is ever increasing.

http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun-1thread/btrfs1.ffsb.random_writes__threads_0001.09-04-08_13.05.54/analysis/iostat-processed.001/chart.html

Looking at graph #8 DM IO/sec, we see that there is even a pattern within the pattern of spikes. It # of IOs in each spike appears to change at each interval and repeats over a set of 7, 30 second intervals.

Quoted 5 lines

Also, we see that we average 12MB/sec of data written out, for 5MB/sec of benchmark throughput.

I have queued up a run without checksums and cow to see how much this overhead is reduced.

Really interesting, thanks Steve.

I'll have to run it at home next week, but I think the high metadata writeback is related to updating backrefs on the extent allocation tree.

Most of the reads during the random write are from the same thing. So, we're experimenting with changes on that end as well.

-chris

#2 lore raw
Quoted 12 lines · Chris Mason
Quoted 10 lines · Steven Pratt

Given the anomalies we were seeing on random write workloads, I decided to simplify the test and do single threaded odirect random write. This should eliminate the locking issue as well as any pdflush bursty behavior. What I got was not quite what I expected.

The most interesting graph is probably #12, DM write throughput. We see a baseline of ~7MB/sec with spikes every 30 seconds. I assume the spike are meta data related as the io is being done from user space at a steady constant rate. The really odd thing is that for the entire almost 2 hour duration, the amplitude of the spike continues to climb, meaning the amount of meta data need to be flushed to disk is ever increasing.

http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun-1thread/btrfs1.ffsb.random_writes__threads_0001.09-04-08_13.05.54/analysis/iostat-processed.001/chart.html

Looking at graph #8 DM IO/sec, we see that there is even a pattern within the pattern of spikes. It # of IOs in each spike appears to change at each interval and repeats over a set of 7, 30 second intervals.

Also, we see that we average 12MB/sec of data written out, for 5MB/sec of benchmark throughput.

Quoted 8 lines
Quoted 2 lines

I have queued up a run without checksums and cow to see how much this overhead is reduced.

Really interesting, thanks Steve.

I'll have to run it at home next week, but I think the high metadata writeback is related to updating backrefs on the extent allocation tree.

Well, looks like you are correct. Using nodatacow has virtually eliminated the extra writes. I is also responsible for a whopping 40x increase in multi threaded random write performance! (2.5MB/sec -> 95MB/sec). See complete details in the new history graphs which I have updated with a new baseline, a run with no csums, and a run with no csums and no cow.

http://btrfs.boxacle.net/repository/raid/history/History.html

nocow make massive differences on the random write workloads, while no csums help the heavily threaded sequential workloads (sequential read and create).

Steve

Quoted 6 lines

Most of the reads during the random write are from the same thing. So, we're experimenting with changes on that end as well.

-chris

#3 lore raw
Quoted 34 lines · Steven Pratt
Quoted 24 lines · Chris Mason
Quoted 22 lines · Steven Pratt

Given the anomalies we were seeing on random write workloads, I decided to simplify the test and do single threaded odirect random write. This should eliminate the locking issue as well as any pdflush bursty behavior. What I got was not quite what I expected.

The most interesting graph is probably #12, DM write throughput. We see a baseline of ~7MB/sec with spikes every 30 seconds. I assume the spike are meta data related as the io is being done from user space at a steady constant rate. The really odd thing is that for the entire almost 2 hour duration, the amplitude of the spike continues to climb, meaning the amount of meta data need to be flushed to disk is ever increasing.

http://btrfs.boxacle.net/repository/raid/longrun/btrfs-longrun-1thread/btrfs1.ffsb.random_writes__threads_0001.09-04-08_13.05.54/analysis/iostat-processed.001/chart.html

Looking at graph #8 DM IO/sec, we see that there is even a pattern within the pattern of spikes. It # of IOs in each spike appears to change at each interval and repeats over a set of 7, 30 second intervals.

Also, we see that we average 12MB/sec of data written out, for 5MB/sec of benchmark throughput.

Quoted 9 lines
Quoted 3 lines

I have queued up a run without checksums and cow to see how much this overhead is reduced.

Really interesting, thanks Steve.

I'll have to run it at home next week, but I think the high metadata writeback is related to updating backrefs on the extent allocation tree.

Quoted 9 lines

Well, looks like you are correct. Using nodatacow has virtually eliminated the extra writes. I is also responsible for a whopping 40x increase in multi threaded random write performance! (2.5MB/sec -> 95MB/sec). See complete details in the new history graphs which I have updated with a new baseline, a run with no csums, and a run with no csums and no cow.

http://btrfs.boxacle.net/repository/raid/history/History.html

Whoa. So, we're on the right track, its good to know the btree locking is scaling well enough for the main btree as well.

Quoted 3 lines

nocow make massive differences on the random write workloads, while no csums help the heavily threaded sequential workloads (sequential read and create).

Ok, thanks again.

-chris