Quick bcache benchmark
I've been very remiss in posting benchmarks; this isn't much, but if anyone has suggestions for what they want I'll see if I can run it.
This is on an old corsair nova - bcache can go something like 10x faster but this is what I have at home. The profile is still interesting, though.
The benchmark is 4k random O_DIRECT reads on a 16 gb file, all in cache
- the idea is to push the b+tree.
Also, the backing device is a md raid10 - so that's working, provided you format your cache with buckets not greater than 1 mb.
root@utumno:/mnt# perf record -afg fio ~/rw4k
randwrite: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio 1.59
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [69914K/0K /s] [17.7K/0 iops] [eta 00m:00s]
randwrite: (groupid=0, jobs=1): err= 0: pid=1247
read : io=16384MB, bw=68713KB/s, iops=17178 , runt=244169msec
cpu : usr=5.66%, sys=22.93%, ctx=4198688, majf=0, minf=85
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued r/w/d: total=4194367/0/0, short=0/0/0Run status group 0 (all jobs):
READ: io=16384MB, aggrb=68712KB/s, minb=70361KB/s, maxb=70361KB/s, mint=244169msec, maxt=244169msec
Disk stats (read/write):
bcache0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
7.74% fio fio [.] 0x1d2ed 5.26% swapper [kernel.kallsyms] [k] ahci_interrupt 3.02% swapper [kernel.kallsyms] [k] mwait_idle 2.52% swapper [kernel.kallsyms] [k] _raw_spin_lock_irqsave 1.82% swapper [kernel.kallsyms] [k] ahci_scr_read 1.68% fio [kernel.kallsyms] [k] __bset_search <- first bcache function 1.37% fio [kernel.kallsyms] [k] __blockdev_direct_IO 1.36% swapper [kernel.kallsyms] [k] irq_entries_start 1.25% swapper [kernel.kallsyms] [k] mix_pool_bytes_extract 1.06% swapper [kernel.kallsyms] [k] kmem_cache_free 0.94% fio [kernel.kallsyms] [k] __switch_to 0.92% fio [kernel.kallsyms] [k] system_call 0.87% swapper [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore 0.84% swapper [kernel.kallsyms] [k] ata_qc_new_init 0.80% fio [kernel.kallsyms] [k] __schedule 0.76% fio [kernel.kallsyms] [k] do_io_submit 0.74% fio [kernel.kallsyms] [k] _raw_spin_lock_irq 0.74% swapper [kernel.kallsyms] [k] _raw_spin_lock 0.73% fio [kernel.kallsyms] [k] ext4_ext_find_extent 0.71% fio [kernel.kallsyms] [k] kmem_cache_alloc 0.70% fio [kernel.kallsyms] [k] read_events 0.65% fio libaio.so.1.0.1 [.] 0x665 0.64% swapper [kernel.kallsyms] [k] __schedule 0.63% swapper [kernel.kallsyms] [k] native_sched_clock 0.63% fio [kernel.kallsyms] [k] btree_search_leaf <- second bcache function 0.62% fio [kernel.kallsyms] [k] bcache_make_request 0.59% swapper [kernel.kallsyms] [k] read_tsc 0.58% fio [kernel.kallsyms] [k] aio_read_evt 0.58% swapper [kernel.kallsyms] [k] select_task_rq_fair 0.57% swapper [kernel.kallsyms] [k] tick_nohz_stop_sched_tick 0.57% fio [kernel.kallsyms] [k] __request_read 0.57% fio [kernel.kallsyms] [k] generic_make_request 0.56% swapper [kernel.kallsyms] [k] sd_prep_fn 0.53% fio [kernel.kallsyms] [k] _raw_spin_lock 0.51% swapper [kernel.kallsyms] [k] __hrtimer_start_range_ns 0.50% fio [kernel.kallsyms] [k] __math_state_restore
Some of the calls to kmem_cache_(free|alloc) are of course from bcache but it looks to be under 25%.
Quoted 1 line · Bostjan Skufca
Random write test?
Sure.
That corsair was giving me /terrible/ write performance, pulled the intel SSD out of my other machine (unregistered the cache from the backing device and attached the new SSD all without unmounting the filesystem :)
Write performance with the intel is not /awesome/, but much more reasonable:
root@utumno:/mnt# perf record -afg fio ~/rw4k
randwrite: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio 1.59
Starting 1 process
Jobs: 1 (f=1): [w] [100.0% done] [0K/98365K /s] [0 /24.2K iops] [eta 00m:00s]
randwrite: (groupid=0, jobs=1): err= 0: pid=1560
write: io=16384MB, bw=89547KB/s, iops=22386 , runt=187359msec
cpu : usr=3.94%, sys=14.82%, ctx=300435, majf=0, minf=19
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued r/w/d: total=0/4194367/0, short=0/0/0Run status group 0 (all jobs):
WRITE: io=16384MB, aggrb=89547KB/s, minb=91696KB/s, maxb=91696KB/s, mint=187359msec, maxt=187359msec
Disk stats (read/write):
bcache0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
8.97% fio fio [.] 0xd1b2 1.64% fio [kernel.kallsyms] [k] bio_insert <- first bcache function 1.56% fio [kernel.kallsyms] [k] __blockdev_direct_IO 1.24% kworker/1:2 [kernel.kallsyms] [k] __bset_search 1.24% kworker/0:0 [kernel.kallsyms] [k] __bset_search 1.19% swapper [kernel.kallsyms] [k] ahci_interrupt 1.17% kworker/0:1 [kernel.kallsyms] [k] __bset_search 1.17% kworker/1:0 [kernel.kallsyms] [k] __bset_search 1.09% fio [kernel.kallsyms] [k] system_call 1.06% kworker/0:2 [kernel.kallsyms] [k] __bset_search 1.06% kworker/1:1 [kernel.kallsyms] [k] __bset_search 1.04% fio [kernel.kallsyms] [k] ext4_ext_find_extent 0.96% fio [kernel.kallsyms] [k] _raw_spin_lock_irq 0.92% fio [kernel.kallsyms] [k] bcache_make_request 0.87% swapper [kernel.kallsyms] [k] _raw_spin_lock_irqsave 0.85% swapper [kernel.kallsyms] [k] mwait_idle 0.83% fio [kernel.kallsyms] [k] do_io_submit 0.77% fio [kernel.kallsyms] [k] memset 0.70% fio [kernel.kallsyms] [k] kmem_cache_alloc 0.65% fio [kernel.kallsyms] [k] md5_transform 0.63% fio [kernel.kallsyms] [k] _raw_spin_lock_irqsave 0.61% fio [kernel.kallsyms] [k] _raw_spin_lock 0.58% fio [kernel.kallsyms] [k] generic_make_request 0.57% fio libaio.so.1.0.1 [.] 0x6b7 0.50% fio [kernel.kallsyms] [k] gup_pte_range
haven't seen bio_insert() show up that high in a profile before, wonder what's up with that..
Reran the random read benchmark with the intel:
root@utumno:/mnt# perf record -afg fio ~/rw4k
randwrite: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio 1.59
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [190.9M/0K /s] [47.7K/0 iops] [eta 00m:00s]
randwrite: (groupid=0, jobs=1): err= 0: pid=1575
read : io=16384MB, bw=153120KB/s, iops=38279 , runt=109571msec
cpu : usr=7.22%, sys=52.15%, ctx=678086, majf=0, minf=85
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued r/w/d: total=4194367/0/0, short=0/0/0Run status group 0 (all jobs):
READ: io=16384MB, aggrb=153119KB/s, minb=156794KB/s, maxb=156794KB/s, mint=109571msec, maxt=109571msec
Disk stats (read/write):
bcache0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
Basically, whatever hardware you have bcache will easily max it out.
Nice, 22k iops for random write is not bad at all (especially compared to spinning disks:)
I have a couple of questions, can you please confirm that I am understanding bcache correctly:
1. When you issue that many random write requests, they get written to SSD first. Then they are slowly propagated from SSD to spinning disk, right? In original order or is the order optimized?2. What about when I unregister bcache from a device? Does it flush changes from SSD to platter?3. Same question (2) for unmounting a drive?4. If machine crashes, will bcache replay changes from SSD to platter at mount time?5. Does it export a number of writes that are pending on SSD, via some /proc or /sys interface?6. Is read cache hot or cold at boot time?
I know that is an overkill for wording "couple of questions", sorry:)
b.
Quoted 88 lines · Kent Overstreet
Quoted 1 line · Bostjan Skufca
Random write test?
Sure.
That corsair was giving me /terrible/ write performance, pulled the intel SSD out of my other machine (unregistered the cache from the backing device and attached the new SSD all without unmounting the filesystem :)
Write performance with the intel is not /awesome/, but much more reasonable:
root@utumno:/mnt# perf record -afg fio ~/rw4k randwrite: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64 fio 1.59 Starting 1 process Jobs: 1 (f=1): [w] [100.0% done] [0K/98365K /s] [0 /24.2K iops] [eta 00m:00s] randwrite: (groupid=0, jobs=1): err= 0: pid=1560 write: io=16384MB, bw=89547KB/s, iops=22386 , runt=187359msec cpu : usr=3.94%, sys=14.82%, ctx=300435, majf=0, minf=19 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued r/w/d: total=0/4194367/0, short=0/0/0Run status group 0 (all jobs):
WRITE: io=16384MB, aggrb=89547KB/s, minb=91696KB/s, maxb=91696KB/s, mint=187359msec, maxt=187359msecDisk stats (read/write):
bcache0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%8.97% fio fio [.]
0xd1b2
1.64% fio [kernel.kallsyms] [k]bio_insert<- first bcache function 1.56% fio [kernel.kallsyms] [k]__blockdev_direct_IO
1.24% kworker/1:2 [kernel.kallsyms] [k]__bset_search
1.24% kworker/0:0 [kernel.kallsyms] [k]__bset_search
1.19% swapper [kernel.kallsyms] [k]ahci_interrupt
1.17% kworker/0:1 [kernel.kallsyms] [k]__bset_search
1.17% kworker/1:0 [kernel.kallsyms] [k]__bset_search
1.09% fio [kernel.kallsyms] [k]system_call
1.06% kworker/0:2 [kernel.kallsyms] [k]__bset_search
1.06% kworker/1:1 [kernel.kallsyms] [k]__bset_search
1.04% fio [kernel.kallsyms] [k]ext4_ext_find_extent
0.96% fio [kernel.kallsyms] [k]_raw_spin_lock_irq
0.92% fio [kernel.kallsyms] [k]bcache_make_request
0.87% swapper [kernel.kallsyms] [k]_raw_spin_lock_irqsave
0.85% swapper [kernel.kallsyms] [k]mwait_idle
0.83% fio [kernel.kallsyms] [k]do_io_submit
0.77% fio [kernel.kallsyms] [k] memset
0.70% fio [kernel.kallsyms] [k]kmem_cache_alloc
0.65% fio [kernel.kallsyms] [k]md5_transform
0.63% fio [kernel.kallsyms] [k]_raw_spin_lock_irqsave
0.61% fio [kernel.kallsyms] [k]_raw_spin_lock
0.58% fio [kernel.kallsyms] [k]generic_make_request
0.57% fio libaio.so.1.0.1 [.]0x6b7
0.50% fio [kernel.kallsyms] [k]gup_pte_rangehaven't seen
bio_insert()show up that high in a profile before, wonder what's up with that..Reran the random read benchmark with the intel:
root@utumno:/mnt# perf record -afg fio ~/rw4k randwrite: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64 fio 1.59 Starting 1 process Jobs: 1 (f=1): [r] [100.0% done] [190.9M/0K /s] [47.7K/0 iops] [eta 00m:00s] randwrite: (groupid=0, jobs=1): err= 0: pid=1575 read : io=16384MB, bw=153120KB/s, iops=38279 , runt=109571msec cpu : usr=7.22%, sys=52.15%, ctx=678086, majf=0, minf=85 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued r/w/d: total=4194367/0/0, short=0/0/0Run status group 0 (all jobs):
READ: io=16384MB, aggrb=153119KB/s, minb=156794KB/s, maxb=156794KB/s, mint=109571msec, maxt=109571msecDisk stats (read/write):
bcache0: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%Basically, whatever hardware you have bcache will easily max it out.
I'm also curious as to how it decides what to keep in cache and whatto toss out, what to write direct to platter and what to buffer. I'vebeen testing LSI's cachecade 2.0 pro, and my intent is to post somebenchmarks between the two. From what I've seen you get at most 1/2performance of your SSD if everything could fit into cache, I'm notsure if that's due to their algorithm and how they decide what's SSDworthy and what's not.
Weird. That wouldn't be blocksize - a tiny bucket size could cause performance issues, but not consistent with what you describe.
Might be some sort of interaction with xfs, I'll have to see if I can reproduce it.
Quoted 18 lines · Marcus Sorensen
Got to try this out quickly this afternoon. Used 200GB hardware raid1 caching for 8 disk, 8T raid 10. Enabled writeback, put xfs on bcache0. Mkfs.xfs took awhile, which was unusual. I mounted the filesystem, created an 8GB file, which was fast. Then ran some 512b random reads against it(16 threads), almost sad speed. Switched same test to random writes, and it was as slow as spindle. Some of the threads even threw "blocked for 120 seconds" traces. I wonder if my blocksize is set wrong on the cache, sort of hard to find the appropriate numbers.
Quoted 8 lines · Marcus Sorensen
I'm also curious as to how it decides what to keep in cache and whatto toss out, what to write direct to platter and what to buffer. I'vebeen testing LSI's cachecade 2.0 pro, and my intent is to post somebenchmarks between the two. From what I've seen you get at most 1/2performance of your SSD if everything could fit into cache, I'm notsure if that's due to their algorithm and how they decide what's SSDworthy and what's not.
Here's some more info. I'm running kernel 3.1.4. When I do random writes, the 'bypassed' number increases in stats. Now I'm random writing direct to /dev/bcache0 and get the same result.
The application I'm using to test does the following:
1. looks at size of test file
2. divides size of file by a cmd line specified io size (tried 512b and 4k) and considers that the blockcount for file
3. randomly selects a block number between 0 and blockcount
4. writes a random string of characters of blocksize to specified block
5. repeat 3 and 4
I'll try a few other benchmark tools.
[root@sansrv2-10 bcache]# for i in ls; do echo -n "$i "; cat $i; done label
readahead 0
running 1sequential_cutoff 4.0Msequential_merge 1
state dirty
verify 0
writeback 1writeback_delay 30writeback_metadata 1writeback_percent 0writeback_running 1
SSD benchmark:
[root@sansrv2-10 ~]# ./seekmark -t16 -q -w destroy-data -f /dev/sde
WRITE benchmarking against /dev/sde 218880 MB
total time: 5.39, time per WRITE request(ms): 0.067 14839.55 total seeks per sec, 927.47 WRITE seeks per sec per thread
bcache0 benchmark:
[root@sansrv2-10 ~]# ./seekmark -t16 -q -w destroy-data -f /dev/bcache0
WRITE benchmarking against /dev/bcache0 7628799 MB
total time: 510.75, time per WRITE request(ms): 6.384 156.63 total seeks per sec, 9.79 WRITE seeks per sec per thread
There also seems to be some work needed with clean-up, since I'm unfamiliar with how bcache works I attempted to make-bcache twice, thinking I'd start over. That worked, but because my cache device was already registered I was unable to re-register my newly formatted cache dev, got "kobject_add_internal failed for bcache with -EEXIST, don't try to register things with the same name in the same directory." I was still able to use my cache device via the old uuid, but this will probably cause problems on reboot. Perhaps an unregister file in /sys/fs/bcache would help, I also tried rmmod'ing bcache to see if I could clear /sys/fs/bcache, but no luck. make-bcache should perhaps check for an existing superblock, ask for confirmation, and give some sort instruction on how to unregister, or do it for you if you reformat.
Quoted 25 lines · Kent Overstreet
Weird. That wouldn't be blocksize - a tiny bucket size could cause performance issues, but not consistent with what you describe.
Might be some sort of interaction with xfs, I'll have to see if I can reproduce it.
Quoted 18 lines · Marcus Sorensen
Got to try this out quickly this afternoon. Used 200GB hardware raid1 caching for 8 disk, 8T raid 10. Enabled writeback, put xfs on bcache0. Mkfs.xfs took awhile, which was unusual. I mounted the filesystem, created an 8GB file, which was fast. Then ran some 512b random reads against it(16 threads), almost sad speed. Switched same test to random writes, and it was as slow as spindle. Some of the threads even threw "blocked for 120 seconds" traces. I wonder if my blocksize is set wrong on the cache, sort of hard to find the appropriate numbers.
Quoted 8 lines · Marcus Sorensen
I'm also curious as to how it decides what to keep in cache and whatto toss out, what to write direct to platter and what to buffer. I'vebeen testing LSI's cachecade 2.0 pro, and my intent is to post somebenchmarks between the two. From what I've seen you get at most 1/2performance of your SSD if everything could fit into cache, I'm notsure if that's due to their algorithm and how they decide what's SSDworthy and what's not.
Oh, and here are my stats after running that write benchmark on bcache0. That's pretty much the only thing I've done in these stats.
[root@sansrv2-10 stats_day]# for i in ls; do echo -n "$i "; cat $i; done 2>/dev/null
bypassed 605Mcache_bypass_hits 333cache_bypass_misses 77553cache_hit_ratio 0cache_hits 85cache_miss_collisions 9256cache_misses 10031cache_readaheads 0
Quoted 97 lines · Marcus Sorensen
Here's some more info. I'm running kernel 3.1.4. When I do random writes, the 'bypassed' number increases in stats. Now I'm random writing direct to
/dev/bcache0and get the same result.The application I'm using to test does the following:
1. looks at size of test file
2. divides size of file by a cmd line specified io size (tried 512b and 4k) and considers that the blockcount for file
3. randomly selects a block number between 0 and blockcount
4. writes a random string of characters of blocksize to specified block
5. repeat 3 and 4I'll try a few other benchmark tools.
[root@sansrv2-10 bcache]# for i in
ls; do echo -n "$i "; cat $i; done label
readahead 0
running 1sequential_cutoff4.0Msequential_merge1
state dirty
verify 0
writeback 1writeback_delay30writeback_metadata1writeback_percent0writeback_running1SSD benchmark:
[root@sansrv2-10 ~]# ./seekmark -t16 -q -w destroy-data -f/dev/sdeWRITE benchmarking against
/dev/sde218880 MBtotal time: 5.39, time per WRITE request(ms): 0.067 14839.55 total seeks per sec, 927.47 WRITE seeks per sec per thread
bcache0 benchmark:
[root@sansrv2-10 ~]# ./seekmark -t16 -q -w destroy-data -f
/dev/bcache0WRITE benchmarking against
/dev/bcache07628799 MBtotal time: 510.75, time per WRITE request(ms): 6.384 156.63 total seeks per sec, 9.79 WRITE seeks per sec per thread
There also seems to be some work needed with clean-up, since I'm unfamiliar with how bcache works I attempted to make-bcache twice, thinking I'd start over. That worked, but because my cache device was already registered I was unable to re-register my newly formatted cache dev, got "
kobject_add_internalfailed for bcache with-EEXIST, don't try to register things with the same name in the same directory." I was still able to use my cache device via the old uuid, but this will probably cause problems on reboot. Perhaps an unregister file in/sys/fs/bcachewould help, I also tried rmmod'ing bcache to see if I could clear/sys/fs/bcache, but no luck. make-bcache should perhaps check for an existing superblock, ask for confirmation, and give some sort instruction on how to unregister, or do it for you if you reformat.Quoted 25 lines · Kent Overstreet
Weird. That wouldn't be blocksize - a tiny bucket size could cause performance issues, but not consistent with what you describe.
Might be some sort of interaction with xfs, I'll have to see if I can reproduce it.
Quoted 18 lines · Marcus Sorensen
Got to try this out quickly this afternoon. Used 200GB hardware raid1 caching for 8 disk, 8T raid 10. Enabled writeback, put xfs on bcache0. Mkfs.xfs took awhile, which was unusual. I mounted the filesystem, created an 8GB file, which was fast. Then ran some 512b random reads against it(16 threads), almost sad speed. Switched same test to random writes, and it was as slow as spindle. Some of the threads even threw "blocked for 120 seconds" traces. I wonder if my blocksize is set wrong on the cache, sort of hard to find the appropriate numbers.
Quoted 8 lines · Marcus Sorensen
I'm also curious as to how it decides what to keep in cache and whatto toss out, what to write direct to platter and what to buffer. I'vebeen testing LSI's cachecade 2.0 pro, and my intent is to post somebenchmarks between the two. From what I've seen you get at most 1/2performance of your SSD if everything could fit into cache, I'm notsure if that's due to their algorithm and how they decide what's SSDworthy and what's not.
Quoted 3 lines · Marcus Sorensen
Here's some more info. I'm running kernel 3.1.4. When I do random writes, the 'bypassed' number increases in stats. Now I'm random writing direct to
/dev/bcache0and get the same result.
Weird. From what you're describing it sounds like throttling is screwed up (and it was recently), but I can't reproduce it now.
Can you try echoing 0 to congested_threshold_us in the cache set dir, and seeing if that fixes it?
Quoted 5 lines
There also seems to be some work needed with clean-up, since I'm unfamiliar with how bcache works I attempted to make-bcache twice, thinking I'd start over. That worked, but because my cache device was already registered I was unable to re-register my newly formatted cache dev, got "
kobject_add_internalfailed for bcache with-EEXIST,
Quoted 8 lines
don't try to register things with the same name in the same directory." I was still able to use my cache device via the old uuid, but this will probably cause problems on reboot. Perhaps an unregister file in
/sys/fs/bcachewould help, I also tried rmmod'ing bcache to see if I could clear/sys/fs/bcache, but no luck. make-bcache should perhaps check for an existing superblock, ask for confirmation, and give some sort instruction on how to unregister, or do it for you if you reformat.
Yeah, I think for some reason bcache isn't opening the devices exclusively on 3.1. I'll have a look...
That keeps the 'bypassed' value from increasing, but it doesn't change write performance.
BEFORE:
[root@sansrv2-10 stats_day]# cat *
27.6M
83
3500
0
166
24380
40660
0
...benchmarking...
AFTER:
[root@sansrv2-10 stats_day]# for i in ls; do echo -n "$i "; cat $i;
Quoted 1 line
done 2>/dev/null
bypassed 27.6Mcache_bypass_hits 83cache_bypass_misses 3500cache_hit_ratio 0cache_hits 410cache_miss_collisions 48879cache_misses 80545cache_readaheads 0
/sys/fs/bcache/60da061c-d646-4ebe-931a-d8580add411d
average_key_size 0block_size 2.0kbtree_cache_size 3.2Mbucket_size 1.0Mcache_available_percent 100clear_stats congested 0congested_threshold_us 0dirty_data 0io_error_halflife 0io_error_limit 8root_usage_percent 0
synchronous 1tree_depth 1
Quoted 27 lines · Kent Overstreet
Quoted 3 lines · Marcus Sorensen
Here's some more info. I'm running kernel 3.1.4. When I do random writes, the 'bypassed' number increases in stats. Now I'm random writing direct to
/dev/bcache0and get the same result.Weird. From what you're describing it sounds like throttling is screwed up (and it was recently), but I can't reproduce it now.
Can you try echoing 0 to
congested_threshold_usin the cache set dir, and seeing if that fixes it?Quoted 5 lines
There also seems to be some work needed with clean-up, since I'm unfamiliar with how bcache works I attempted to make-bcache twice, thinking I'd start over. That worked, but because my cache device was already registered I was unable to re-register my newly formatted cache dev, got "
kobject_add_internalfailed for bcache with-EEXIST,Quoted 8 lines
don't try to register things with the same name in the same directory." I was still able to use my cache device via the old uuid, but this will probably cause problems on reboot. Perhaps an unregister file in
/sys/fs/bcachewould help, I also tried rmmod'ing bcache to see if I could clear/sys/fs/bcache, but no luck. make-bcache should perhaps check for an existing superblock, ask for confirmation, and give some sort instruction on how to unregister, or do it for you if you reformat.Yeah, I think for some reason bcache isn't opening the devices exclusively on 3.1. I'll have a look...
Any ideas on this? Do you think it's a bug, or am I just holding it wrong? :-)
Quoted 75 lines · Marcus Sorensen
That keeps the 'bypassed' value from increasing, but it doesn't change write performance.
BEFORE:
[root@sansrv2-10 stats_day]# cat *
27.6M
83
3500
0
166
24380
40660
0...benchmarking...
AFTER:
[root@sansrv2-10 stats_day]# for i in
ls; do echo -n "$i "; cat $i;Quoted 1 line
done 2>/dev/null
bypassed 27.6M
cache_bypass_hits83cache_bypass_misses3500cache_hit_ratio0cache_hits410cache_miss_collisions48879cache_misses80545cache_readaheads0
/sys/fs/bcache/60da061c-d646-4ebe-931a-d8580add411d
average_key_size0block_size2.0kbtree_cache_size3.2Mbucket_size1.0Mcache_available_percent100clear_statscongested 0congested_threshold_us0dirty_data0io_error_halflife0io_error_limit8root_usage_percent0
synchronous 1tree_depth1Quoted 27 lines · Kent Overstreet
Quoted 3 lines · Marcus Sorensen
Here's some more info. I'm running kernel 3.1.4. When I do random writes, the 'bypassed' number increases in stats. Now I'm random writing direct to
/dev/bcache0and get the same result.Weird. From what you're describing it sounds like throttling is screwed up (and it was recently), but I can't reproduce it now.
Can you try echoing 0 to
congested_threshold_usin the cache set dir, and seeing if that fixes it?Quoted 5 lines
There also seems to be some work needed with clean-up, since I'm unfamiliar with how bcache works I attempted to make-bcache twice, thinking I'd start over. That worked, but because my cache device was already registered I was unable to re-register my newly formatted cache dev, got "
kobject_add_internalfailed for bcache with-EEXIST,Quoted 8 lines
don't try to register things with the same name in the same directory." I was still able to use my cache device via the old uuid, but this will probably cause problems on reboot. Perhaps an unregister file in
/sys/fs/bcachewould help, I also tried rmmod'ing bcache to see if I could clear/sys/fs/bcache, but no luck. make-bcache should perhaps check for an existing superblock, ask for confirmation, and give some sort instruction on how to unregister, or do it for you if you reformat.Yeah, I think for some reason bcache isn't opening the devices exclusively on 3.1. I'll have a look...
Sorry, I was thinking about that issue for awhile and then I got distracted...
It's not user error, it's an irritating corner case. Basically, it's the result of a workaround for a particularly obscure data corruption bug.
If a write bypasses the cache, it has to invalidate that region of the cache; the null key it leaves in the cache will block cache misses from adding that data to the cache until the btree node fills up (and possibly splits).
It hasn't been an issue for us in normal operation, but when you're just testing - i.e. you don't have much load - that node split may not happen for a long time, and so if for some reason a bunch of data bypassed the cache... well, you see what happens.
Unfortunately a better solution to the original race is not going to be simple, so it's probably not going to be done in the very near future. It's a _very_ difficult race to hit, but in the meantime I'd rather lose performance than corrupt data.
But the good news is if you put normal server-ish load on it the issue should go away in steady state operation.
Quoted 82 lines · Marcus Sorensen
Any ideas on this? Do you think it's a bug, or am I just holding it wrong? :-)
Quoted 46 lines · Marcus Sorensen
That keeps the 'bypassed' value from increasing, but it doesn't change write performance.
BEFORE:
[root@sansrv2-10 stats_day]# cat *
27.6M
83
3500
0
166
24380
40660
0...benchmarking...
AFTER:
[root@sansrv2-10 stats_day]# for i in
ls; do echo -n "$i "; cat $i;Quoted 1 line
done 2>/dev/null
bypassed 27.6M
cache_bypass_hits83cache_bypass_misses3500cache_hit_ratio0cache_hits410cache_miss_collisions48879cache_misses80545cache_readaheads0
/sys/fs/bcache/60da061c-d646-4ebe-931a-d8580add411d
average_key_size0block_size2.0kbtree_cache_size3.2Mbucket_size1.0Mcache_available_percent100clear_statscongested 0congested_threshold_us0dirty_data0io_error_halflife0io_error_limit8root_usage_percent0
synchronous 1tree_depth1Quoted 29 lines
Quoted 27 lines · Kent Overstreet
Quoted 3 lines · Marcus Sorensen
Here's some more info. I'm running kernel 3.1.4. When I do random writes, the 'bypassed' number increases in stats. Now I'm random writing direct to
/dev/bcache0and get the same result.Weird. From what you're describing it sounds like throttling is screwed up (and it was recently), but I can't reproduce it now.
Can you try echoing 0 to
congested_threshold_usin the cache set dir, and seeing if that fixes it?Quoted 5 lines
There also seems to be some work needed with clean-up, since I'm unfamiliar with how bcache works I attempted to make-bcache twice, thinking I'd start over. That worked, but because my cache device was already registered I was unable to re-register my newly formatted cache dev, got "
kobject_add_internalfailed for bcache with-EEXIST,Quoted 8 lines
don't try to register things with the same name in the same directory." I was still able to use my cache device via the old uuid, but this will probably cause problems on reboot. Perhaps an unregister file in
/sys/fs/bcachewould help, I also tried rmmod'ing bcache to see if I could clear/sys/fs/bcache, but no luck. make-bcache should perhaps check for an existing superblock, ask for confirmation, and give some sort instruction on how to unregister, or do it for you if you reformat.Yeah, I think for some reason bcache isn't opening the devices exclusively on 3.1. I'll have a look...
--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Thanks! I'll put it through some more tests. I kind of figured that something more real-world would help.
Quoted 107 lines · Kent Overstreet
Sorry, I was thinking about that issue for awhile and then I got distracted...
It's not user error, it's an irritating corner case. Basically, it's the result of a workaround for a particularly obscure data corruption bug.
If a write bypasses the cache, it has to invalidate that region of the cache; the null key it leaves in the cache will block cache misses from adding that data to the cache until the btree node fills up (and possibly splits).
It hasn't been an issue for us in normal operation, but when you're just testing - i.e. you don't have much load - that node split may not happen for a long time, and so if for some reason a bunch of data bypassed the cache... well, you see what happens.
Unfortunately a better solution to the original race is not going to be simple, so it's probably not going to be done in the very near future. It's a _very_ difficult race to hit, but in the meantime I'd rather lose performance than corrupt data.
But the good news is if you put normal server-ish load on it the issue should go away in steady state operation.
Quoted 82 lines · Marcus Sorensen
Any ideas on this? Do you think it's a bug, or am I just holding it wrong? :-)
Quoted 46 lines · Marcus Sorensen
That keeps the 'bypassed' value from increasing, but it doesn't change write performance.
BEFORE:
[root@sansrv2-10 stats_day]# cat *
27.6M
83
3500
0
166
24380
40660
0...benchmarking...
AFTER:
[root@sansrv2-10 stats_day]# for i in
ls; do echo -n "$i "; cat $i;Quoted 1 line
done 2>/dev/null
bypassed 27.6M
cache_bypass_hits83cache_bypass_misses3500cache_hit_ratio0cache_hits410cache_miss_collisions48879cache_misses80545cache_readaheads0
/sys/fs/bcache/60da061c-d646-4ebe-931a-d8580add411d
average_key_size0block_size2.0kbtree_cache_size3.2Mbucket_size1.0Mcache_available_percent100clear_statscongested 0congested_threshold_us0dirty_data0io_error_halflife0io_error_limit8root_usage_percent0
synchronous 1tree_depth1Quoted 29 lines
Quoted 27 lines · Kent Overstreet
Quoted 3 lines · Marcus Sorensen
Here's some more info. I'm running kernel 3.1.4. When I do random writes, the 'bypassed' number increases in stats. Now I'm random writing direct to
/dev/bcache0and get the same result.Weird. From what you're describing it sounds like throttling is screwed up (and it was recently), but I can't reproduce it now.
Can you try echoing 0 to
congested_threshold_usin the cache set dir, and seeing if that fixes it?Quoted 5 lines
There also seems to be some work needed with clean-up, since I'm unfamiliar with how bcache works I attempted to make-bcache twice, thinking I'd start over. That worked, but because my cache device was already registered I was unable to re-register my newly formatted cache dev, got "
kobject_add_internalfailed for bcache with-EEXIST,Quoted 8 lines
don't try to register things with the same name in the same directory." I was still able to use my cache device via the old uuid, but this will probably cause problems on reboot. Perhaps an unregister file in
/sys/fs/bcachewould help, I also tried rmmod'ing bcache to see if I could clear/sys/fs/bcache, but no luck. make-bcache should perhaps check for an existing superblock, ask for confirmation, and give some sort instruction on how to unregister, or do it for you if you reformat.Yeah, I think for some reason bcache isn't opening the devices exclusively on 3.1. I'll have a look...
--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Actually I think this IS user error. I ran a benchmark with FIO, and the results were practically identical with and without bcache. I applied the 3.1.4 kernel patch on top of your 3.1 tree, even though it applied cleanly I'm guessing that wiped something out. Here are my stats after running the benchmark on bcache, and also included is the fio config.
bypassed 32.1Gcache_bypass_hits 5482cache_bypass_misses 194862cache_hit_ratio 3cache_hits 786cache_miss_collisions 206cache_misses 19447cache_readaheads 0
[global]
ioengine=libaio
iodepth=4
invalidate=1 #make sure we're not cached locally direct=1 #don't use buffers during test (test without local caches) thread
ramp_time=20time_based
runtime=180
[8RandomReadWriters]
rw=randrw
numjobs=8
blocksize=4k
size=1G
[2SequentialReadWriters]
rw=rw
numjobs=2
size=4G
blocksize_range=64k-1M
Quoted 111 lines · Marcus Sorensen
Thanks! I'll put it through some more tests. I kind of figured that something more real-world would help.
Quoted 107 lines · Kent Overstreet
Sorry, I was thinking about that issue for awhile and then I got distracted...
It's not user error, it's an irritating corner case. Basically, it's the result of a workaround for a particularly obscure data corruption bug.
If a write bypasses the cache, it has to invalidate that region of the cache; the null key it leaves in the cache will block cache misses from adding that data to the cache until the btree node fills up (and possibly splits).
It hasn't been an issue for us in normal operation, but when you're just testing - i.e. you don't have much load - that node split may not happen for a long time, and so if for some reason a bunch of data bypassed the cache... well, you see what happens.
Unfortunately a better solution to the original race is not going to be simple, so it's probably not going to be done in the very near future. It's a _very_ difficult race to hit, but in the meantime I'd rather lose performance than corrupt data.
But the good news is if you put normal server-ish load on it the issue should go away in steady state operation.
Quoted 82 lines · Marcus Sorensen
Any ideas on this? Do you think it's a bug, or am I just holding it wrong? :-)
Quoted 46 lines · Marcus Sorensen
That keeps the 'bypassed' value from increasing, but it doesn't change write performance.
BEFORE:
[root@sansrv2-10 stats_day]# cat *
27.6M
83
3500
0
166
24380
40660
0...benchmarking...
AFTER:
[root@sansrv2-10 stats_day]# for i in
ls; do echo -n "$i "; cat $i;Quoted 1 line
done 2>/dev/null
bypassed 27.6M
cache_bypass_hits83cache_bypass_misses3500cache_hit_ratio0cache_hits410cache_miss_collisions48879cache_misses80545cache_readaheads0
/sys/fs/bcache/60da061c-d646-4ebe-931a-d8580add411d
average_key_size0block_size2.0kbtree_cache_size3.2Mbucket_size1.0Mcache_available_percent100clear_statscongested 0congested_threshold_us0dirty_data0io_error_halflife0io_error_limit8root_usage_percent0
synchronous 1tree_depth1Quoted 29 lines
Quoted 27 lines · Kent Overstreet
Quoted 3 lines · Marcus Sorensen
Here's some more info. I'm running kernel 3.1.4. When I do random writes, the 'bypassed' number increases in stats. Now I'm random writing direct to
/dev/bcache0and get the same result.Weird. From what you're describing it sounds like throttling is screwed up (and it was recently), but I can't reproduce it now.
Can you try echoing 0 to
congested_threshold_usin the cache set dir, and seeing if that fixes it?Quoted 5 lines
There also seems to be some work needed with clean-up, since I'm unfamiliar with how bcache works I attempted to make-bcache twice, thinking I'd start over. That worked, but because my cache device was already registered I was unable to re-register my newly formatted cache dev, got "
kobject_add_internalfailed for bcache with-EEXIST,Quoted 8 lines
don't try to register things with the same name in the same directory." I was still able to use my cache device via the old uuid, but this will probably cause problems on reboot. Perhaps an unregister file in
/sys/fs/bcachewould help, I also tried rmmod'ing bcache to see if I could clear/sys/fs/bcache, but no luck. make-bcache should perhaps check for an existing superblock, ask for confirmation, and give some sort instruction on how to unregister, or do it for you if you reformat.Yeah, I think for some reason bcache isn't opening the devices exclusively on 3.1. I'll have a look...
--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
That's what you'd expect in writethrough mode when you aren't getting any cache hits - try flipping on writeback and see what happens.
Quoted 151 lines · Marcus Sorensen
Actually I think this IS user error. I ran a benchmark with FIO, and the results were practically identical with and without bcache. I applied the 3.1.4 kernel patch on top of your 3.1 tree, even though it applied cleanly I'm guessing that wiped something out. Here are my stats after running the benchmark on bcache, and also included is the fio config.
bypassed 32.1G
cache_bypass_hits5482cache_bypass_misses194862cache_hit_ratio3cache_hits786cache_miss_collisions206cache_misses19447cache_readaheads0[global]
ioengine=libaio
iodepth=4
invalidate=1 #make sure we're not cached locally direct=1 #don't use buffers during test (test without local caches) thread
ramp_time=20time_based
runtime=180[8RandomReadWriters]
rw=randrw
numjobs=8
blocksize=4k
size=1G[2SequentialReadWriters]
rw=rw
numjobs=2
size=4G
blocksize_range=64k-1MQuoted 111 lines · Marcus Sorensen
Thanks! I'll put it through some more tests. I kind of figured that something more real-world would help.
Quoted 107 lines · Kent Overstreet
Sorry, I was thinking about that issue for awhile and then I got distracted...
It's not user error, it's an irritating corner case. Basically, it's the result of a workaround for a particularly obscure data corruption bug.
If a write bypasses the cache, it has to invalidate that region of the cache; the null key it leaves in the cache will block cache misses from adding that data to the cache until the btree node fills up (and possibly splits).
It hasn't been an issue for us in normal operation, but when you're just testing - i.e. you don't have much load - that node split may not happen for a long time, and so if for some reason a bunch of data bypassed the cache... well, you see what happens.
Unfortunately a better solution to the original race is not going to be simple, so it's probably not going to be done in the very near future. It's a _very_ difficult race to hit, but in the meantime I'd rather lose performance than corrupt data.
But the good news is if you put normal server-ish load on it the issue should go away in steady state operation.
Quoted 82 lines · Marcus Sorensen
Any ideas on this? Do you think it's a bug, or am I just holding it wrong? :-)
Quoted 46 lines · Marcus Sorensen
That keeps the 'bypassed' value from increasing, but it doesn't change write performance.
BEFORE:
[root@sansrv2-10 stats_day]# cat *
27.6M
83
3500
0
166
24380
40660
0...benchmarking...
AFTER:
[root@sansrv2-10 stats_day]# for i in
ls; do echo -n "$i "; cat $i;Quoted 1 line
done 2>/dev/null
bypassed 27.6M
cache_bypass_hits83cache_bypass_misses3500cache_hit_ratio0cache_hits410cache_miss_collisions48879cache_misses80545cache_readaheads0
/sys/fs/bcache/60da061c-d646-4ebe-931a-d8580add411d
average_key_size0block_size2.0kbtree_cache_size3.2Mbucket_size1.0Mcache_available_percent100clear_statscongested 0congested_threshold_us0dirty_data0io_error_halflife0io_error_limit8root_usage_percent0
synchronous 1tree_depth1Quoted 29 lines
Quoted 27 lines · Kent Overstreet
Quoted 3 lines · Marcus Sorensen
Here's some more info. I'm running kernel 3.1.4. When I do random writes, the 'bypassed' number increases in stats. Now I'm random writing direct to
/dev/bcache0and get the same result.Weird. From what you're describing it sounds like throttling is screwed up (and it was recently), but I can't reproduce it now.
Can you try echoing 0 to
congested_threshold_usin the cache set dir, and seeing if that fixes it?Quoted 5 lines
There also seems to be some work needed with clean-up, since I'm unfamiliar with how bcache works I attempted to make-bcache twice, thinking I'd start over. That worked, but because my cache device was already registered I was unable to re-register my newly formatted cache dev, got "
kobject_add_internalfailed for bcache with-EEXIST,Quoted 8 lines
don't try to register things with the same name in the same directory." I was still able to use my cache device via the old uuid, but this will probably cause problems on reboot. Perhaps an unregister file in
/sys/fs/bcachewould help, I also tried rmmod'ing bcache to see if I could clear/sys/fs/bcache, but no luck. make-bcache should perhaps check for an existing superblock, ask for confirmation, and give some sort instruction on how to unregister, or do it for you if you reformat.Yeah, I think for some reason bcache isn't opening the devices exclusively on 3.1. I'll have a look...
--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Yeah, I echoed 1 into writeback before doing the test. And why wouldn't I get any cache hits?
Quoted 155 lines · Kent Overstreet
That's what you'd expect in writethrough mode when you aren't getting any cache hits - try flipping on writeback and see what happens.
Quoted 151 lines · Marcus Sorensen
Actually I think this IS user error. I ran a benchmark with FIO, and the results were practically identical with and without bcache. I applied the 3.1.4 kernel patch on top of your 3.1 tree, even though it applied cleanly I'm guessing that wiped something out. Here are my stats after running the benchmark on bcache, and also included is the fio config.
bypassed 32.1G
cache_bypass_hits5482cache_bypass_misses194862cache_hit_ratio3cache_hits786cache_miss_collisions206cache_misses19447cache_readaheads0[global]
ioengine=libaio
iodepth=4
invalidate=1 #make sure we're not cached locally direct=1 #don't use buffers during test (test without local caches) thread
ramp_time=20time_based
runtime=180[8RandomReadWriters]
rw=randrw
numjobs=8
blocksize=4k
size=1G[2SequentialReadWriters]
rw=rw
numjobs=2
size=4G
blocksize_range=64k-1MQuoted 111 lines · Marcus Sorensen
Thanks! I'll put it through some more tests. I kind of figured that something more real-world would help.
Quoted 107 lines · Kent Overstreet
Sorry, I was thinking about that issue for awhile and then I got distracted...
It's not user error, it's an irritating corner case. Basically, it's the result of a workaround for a particularly obscure data corruption bug.
If a write bypasses the cache, it has to invalidate that region of the cache; the null key it leaves in the cache will block cache misses from adding that data to the cache until the btree node fills up (and possibly splits).
It hasn't been an issue for us in normal operation, but when you're just testing - i.e. you don't have much load - that node split may not happen for a long time, and so if for some reason a bunch of data bypassed the cache... well, you see what happens.
Unfortunately a better solution to the original race is not going to be simple, so it's probably not going to be done in the very near future. It's a _very_ difficult race to hit, but in the meantime I'd rather lose performance than corrupt data.
But the good news is if you put normal server-ish load on it the issue should go away in steady state operation.
Quoted 82 lines · Marcus Sorensen
Any ideas on this? Do you think it's a bug, or am I just holding it wrong? :-)
Quoted 46 lines · Marcus Sorensen
That keeps the 'bypassed' value from increasing, but it doesn't change write performance.
BEFORE:
[root@sansrv2-10 stats_day]# cat *
27.6M
83
3500
0
166
24380
40660
0...benchmarking...
AFTER:
[root@sansrv2-10 stats_day]# for i in
ls; do echo -n "$i "; cat $i;Quoted 1 line
done 2>/dev/null
bypassed 27.6M
cache_bypass_hits83cache_bypass_misses3500cache_hit_ratio0cache_hits410cache_miss_collisions48879cache_misses80545cache_readaheads0
/sys/fs/bcache/60da061c-d646-4ebe-931a-d8580add411d
average_key_size0block_size2.0kbtree_cache_size3.2Mbucket_size1.0Mcache_available_percent100clear_statscongested 0congested_threshold_us0dirty_data0io_error_halflife0io_error_limit8root_usage_percent0
synchronous 1tree_depth1Quoted 29 lines
Quoted 27 lines · Kent Overstreet
Quoted 3 lines · Marcus Sorensen
Here's some more info. I'm running kernel 3.1.4. When I do random writes, the 'bypassed' number increases in stats. Now I'm random writing direct to
/dev/bcache0and get the same result.Weird. From what you're describing it sounds like throttling is screwed up (and it was recently), but I can't reproduce it now.
Can you try echoing 0 to
congested_threshold_usin the cache set dir, and seeing if that fixes it?Quoted 5 lines
There also seems to be some work needed with clean-up, since I'm unfamiliar with how bcache works I attempted to make-bcache twice, thinking I'd start over. That worked, but because my cache device was already registered I was unable to re-register my newly formatted cache dev, got "
kobject_add_internalfailed for bcache with-EEXIST,Quoted 8 lines
don't try to register things with the same name in the same directory." I was still able to use my cache device via the old uuid, but this will probably cause problems on reboot. Perhaps an unregister file in
/sys/fs/bcachewould help, I also tried rmmod'ing bcache to see if I could clear/sys/fs/bcache, but no luck. make-bcache should perhaps check for an existing superblock, ask for confirmation, and give some sort instruction on how to unregister, or do it for you if you reformat.Yeah, I think for some reason bcache isn't opening the devices exclusively on 3.1. I'll have a look...
--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Sounds like the cache isn't getting populated for some reason. Have you tried disabling throttling? echo 0 > congested_threshhold_us
With that off and writeback on you really ought to get some performance improvement on random writes...
Quoted 160 lines · Marcus Sorensen
Yeah, I echoed 1 into writeback before doing the test. And why wouldn't I get any cache hits?
Quoted 155 lines · Kent Overstreet
That's what you'd expect in writethrough mode when you aren't getting any cache hits - try flipping on writeback and see what happens.
Quoted 151 lines · Marcus Sorensen
Actually I think this IS user error. I ran a benchmark with FIO, and the results were practically identical with and without bcache. I applied the 3.1.4 kernel patch on top of your 3.1 tree, even though it applied cleanly I'm guessing that wiped something out. Here are my stats after running the benchmark on bcache, and also included is the fio config.
bypassed 32.1G
cache_bypass_hits5482cache_bypass_misses194862cache_hit_ratio3cache_hits786cache_miss_collisions206cache_misses19447cache_readaheads0[global]
ioengine=libaio
iodepth=4
invalidate=1 #make sure we're not cached locally direct=1 #don't use buffers during test (test without local caches) thread
ramp_time=20time_based
runtime=180[8RandomReadWriters]
rw=randrw
numjobs=8
blocksize=4k
size=1G[2SequentialReadWriters]
rw=rw
numjobs=2
size=4G
blocksize_range=64k-1MQuoted 111 lines · Marcus Sorensen
Thanks! I'll put it through some more tests. I kind of figured that something more real-world would help.
Quoted 107 lines · Kent Overstreet
Sorry, I was thinking about that issue for awhile and then I got distracted...
It's not user error, it's an irritating corner case. Basically, it's the result of a workaround for a particularly obscure data corruption bug.
If a write bypasses the cache, it has to invalidate that region of the cache; the null key it leaves in the cache will block cache misses from adding that data to the cache until the btree node fills up (and possibly splits).
It hasn't been an issue for us in normal operation, but when you're just testing - i.e. you don't have much load - that node split may not happen for a long time, and so if for some reason a bunch of data bypassed the cache... well, you see what happens.
Unfortunately a better solution to the original race is not going to be simple, so it's probably not going to be done in the very near future. It's a _very_ difficult race to hit, but in the meantime I'd rather lose performance than corrupt data.
But the good news is if you put normal server-ish load on it the issue should go away in steady state operation.
Quoted 82 lines · Marcus Sorensen
Any ideas on this? Do you think it's a bug, or am I just holding it wrong? :-)
Quoted 46 lines · Marcus Sorensen
That keeps the 'bypassed' value from increasing, but it doesn't change write performance.
BEFORE:
[root@sansrv2-10 stats_day]# cat *
27.6M
83
3500
0
166
24380
40660
0...benchmarking...
AFTER:
[root@sansrv2-10 stats_day]# for i in
ls; do echo -n "$i "; cat $i;Quoted 1 line
done 2>/dev/null
bypassed 27.6M
cache_bypass_hits83cache_bypass_misses3500cache_hit_ratio0cache_hits410cache_miss_collisions48879cache_misses80545cache_readaheads0
/sys/fs/bcache/60da061c-d646-4ebe-931a-d8580add411d
average_key_size0block_size2.0kbtree_cache_size3.2Mbucket_size1.0Mcache_available_percent100clear_statscongested 0congested_threshold_us0dirty_data0io_error_halflife0io_error_limit8root_usage_percent0
synchronous 1tree_depth1Quoted 29 lines
Quoted 27 lines · Kent Overstreet
Quoted 3 lines · Marcus Sorensen
Here's some more info. I'm running kernel 3.1.4. When I do random writes, the 'bypassed' number increases in stats. Now I'm random writing direct to
/dev/bcache0and get the same result.Weird. From what you're describing it sounds like throttling is screwed up (and it was recently), but I can't reproduce it now.
Can you try echoing 0 to
congested_threshold_usin the cache set dir, and seeing if that fixes it?Quoted 5 lines
There also seems to be some work needed with clean-up, since I'm unfamiliar with how bcache works I attempted to make-bcache twice, thinking I'd start over. That worked, but because my cache device was already registered I was unable to re-register my newly formatted cache dev, got "
kobject_add_internalfailed for bcache with-EEXIST,Quoted 8 lines
don't try to register things with the same name in the same directory." I was still able to use my cache device via the old uuid, but this will probably cause problems on reboot. Perhaps an unregister file in
/sys/fs/bcachewould help, I also tried rmmod'ing bcache to see if I could clear/sys/fs/bcache, but no luck. make-bcache should perhaps check for an existing superblock, ask for confirmation, and give some sort instruction on how to unregister, or do it for you if you reformat.Yeah, I think for some reason bcache isn't opening the devices exclusively on 3.1. I'll have a look...
--
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html