ceph on btrfs [was Re: ceph on non-btrfs file systems]

Quoted 22 lines · Christian Brunner

2011/10/26 Sage Weil <[email protected]>:

Quoted 21 lines · Christian Brunner

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Christian, have you tweaked those settings in your ceph.conf? It would be something like 'journal dio = false'. If not, can you verify that directio shows true when the journal is initialized from your osd log? E.g.,

 2011-10-21 15:21:02.026789 7ff7e5c54720 journal _open dev/osd0.journal fd 14: 104857600 bytes, block size 4096 bytes, directio = 1

If directio = 1 for you, something else funky is causing those blkdev_fsync's...

Quoted 8 lines

Quoted 7 lines

Quoted 5 lines

I've looked it up in the logs - directio is 1:

Oct 25 17:20:16 os00 osd.000[1696]: 7f0016841740 journal _open

/dev/vg01/lv_osd_journal_0 fd 15: 17179869184 bytes, block size 4096 bytes, directio = 1

Do you mind capturing an strace? I'd like to see where that blkdev_fsync is coming from.

Here is an strace. I can see a lot of sync_file_range operations.

Yeah, these all look like the flusher thread, and shouldn't be hitting blkdev_fsync. Can you confirm that with

Quoted 6 lines

Quoted 5 lines

filestore flusher = false
filestore sync flush = false

you get no sync_file_range at all? I wonder if this is also perf lying about the call chain.

Yes, setting this makes the sync_file_range calls go away.

Okay. That means either sync_file_range on a regular btrfs file is triggering blkdev_fsync somewhere in btrfs, there is an extremely sneaky bug that is mixing up file descriptors, or latencytop is lying. I'm guessing the latter, given the other weirdness Josef and Chris were seeing. :)

Quoted 1 line

Is it safe to use these settings with "filestore btrfs snap = 0"?

Yeah. They're purely a performance thing to push as much dirty data to disk as quickly as possible to minimize the snapshot create latency. You'll notice the write throughput tends to tank when them off.

sage

Hi
resend without the perf attachment, which could be found here:
http://tuxadero.com/multistorage/perf.report.txt.bz2

Best Regards,
 martin

-------- Original-Nachricht --------
Betreff: Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]
Datum: Wed, 26 Oct 2011 22:38:47 +0200
Von: Martin Mailand <[email protected]>
Antwort an: [email protected]
An: Sage Weil <[email protected]>
Kopie (CC): Christian Brunner <[email protected]>, [email protected], [email protected]

Hi,
I have more or less the same setup as Christian and I suffer the same problems.
But as far as I can see the output of latencytop and perf differs form Christian one, both are attached.
I was wondering about the high latency from btrfs-submit.

Process btrfs-submit-0 (970) Total: 2123.5 msec

I have as well the high IO rate and high IO wait.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle 0.60    0.00    2.20   82.40    0.00   14.80

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util sda               0.00     0.00    0.00    8.40     0.00    74.40 17.71     0.03    3.81    0.00    3.81   3.81   3.20 sdb               0.00     7.00    0.00  269.80     0.00  1224.80 9.08   107.19  398.69    0.00  398.69   3.15  85.00

top - 21:57:41 up  8:41,  1 user,  load average: 0.65, 0.79, 0.76
Tasks: 179 total,   1 running, 178 sleeping,   0 stopped,   0 zombie Cpu(s):  0.6%us,  2.4%sy,  0.0%ni, 70.8%id, 25.8%wa,  0.0%hi,  0.3%si, 0.0%st
Mem:   4018276k total,  1577728k used,  2440548k free,    10496k buffers
Swap:  1998844k total,        0k used,  1998844k free,  1316696k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

 1399 root      20   0  548m 103m 3428 S  0.0  2.6   2:01.85 ceph-osd

 1401 root      20   0  548m 103m 3428 S  0.0  2.6   1:51.71 ceph-osd

 1400 root      20   0  548m 103m 3428 S  0.0  2.6   1:50.30 ceph-osd

 1391 root      20   0     0    0    0 S  0.0  0.0   1:18.39 btrfs-endio-wri

  976 root      20   0     0    0    0 S  0.0  0.0   1:18.11 btrfs-endio-wri

 1367 root      20   0     0    0    0 S  0.0  0.0   1:05.60 btrfs-worker-1

  968 root      20   0     0    0    0 S  0.0  0.0   1:05.45 btrfs-worker-0

 1163 root      20   0  141m 1636 1100 S  0.0  0.0   1:00.56 collectd

  970 root      20   0     0    0    0 S  0.0  0.0   0:47.73 btrfs-submit-0

 1402 root      20   0  548m 103m 3428 S  0.0  2.6   0:34.86 ceph-osd

 1392 root      20   0     0    0    0 S  0.0  0.0   0:33.70 btrfs-endio-met

  975 root      20   0     0    0    0 S  0.0  0.0   0:32.70 btrfs-endio-met

 1415 root      20   0  548m 103m 3428 S  0.0  2.6   0:28.29 ceph-osd

 1414 root      20   0  548m 103m 3428 S  0.0  2.6   0:28.24 ceph-osd

 1397 root      20   0  548m 103m 3428 S  0.0  2.6   0:24.60 ceph-osd

 1436 root      20   0  548m 103m 3428 S  0.0  2.6   0:13.31 ceph-osd

Here ist my setup.
Kernel v3.1 + Josef

The config for this osd (ceph version 0.37 (commit:a6f3bbb744a6faea95ae48317f0b838edb16a896)) is:
[osd.1]
        host = s-brick-003
        osd journal = /dev/sda7
        btrfs devs = /dev/sdb
       btrfs options = noatime filestore_btrfs_snap = false

I hope this helps to pin point the problem.

Best Regards,
martin

Quoted 56 lines · Sage Weil

Quoted 33 lines · Christian Brunner

2011/10/26 Sage Weil <[email protected]>:

Quoted 31 lines · Christian Brunner

Quoted 20 lines

Quoted 20 lines

Quoted 19 lines

Quoted 8 lines

Christian, have you tweaked those settings in your ceph.conf?  It would be
something like 'journal dio = false'.  If not, can you verify that directio shows true when the journal is initialized from your osd log?
E.g.,

Quoted 5 lines

 2011-10-21 15:21:02.026789 7ff7e5c54720 journal _open

dev/osd0.journal fd 14: 104857600 bytes, block size 4096 bytes, directio = 1

If directio = 1 for you, something else funky is causing those blkdev_fsync's...

I've looked it up in the logs - directio is 1:

Oct 25 17:20:16 os00 osd.000[1696]: 7f0016841740 journal _open

/dev/vg01/lv_osd_journal_0 fd 15: 17179869184 bytes, block size 4096 bytes, directio = 1

Quoted 5 lines

Quoted 3 lines

Do you mind capturing an strace?  I'd like to see where that blkdev_fsync
is coming from.

Here is an strace. I can see a lot of sync_file_range operations.

Yeah, these all look like the flusher thread, and shouldn't be hitting blkdev_fsync.  Can you confirm that with

Quoted 7 lines

Quoted 5 lines

      filestore flusher = false
      filestore sync flush = false

you get no sync_file_range at all?  I wonder if this is also perf lying about the call chain.

Yes, setting this makes the sync_file_range calls go away.

Okay.  That means either sync_file_range on a regular btrfs file is triggering blkdev_fsync somewhere in btrfs, there is an extremely sneaky bug that is mixing up file descriptors, or latencytop is lying.  I'm guessing the latter, given the other weirdness Josef and Chris were seeing.  :)

Quoted 1 line

Is it safe to use these settings with "filestore btrfs snap = 0"?

Yeah.  They're purely a performance thing to push as much dirty data to disk as quickly as possible to minimize the snapshot create latency. You'll notice the write throughput tends to tank when them off.

sage

Hi Martin,

a quick dig into your perf report show a large amount of swapper work. If this is the case, i would suspect latency. So do you have not enough physical ram in your machine ?

Greetings

Stefan Majer

Quoted 163 lines · Martin Mailand

Hi
resend without the perf attachment, which could be found here:
http://tuxadero.com/multistorage/perf.report.txt.bz2

Best Regards,
martin

-------- Original-Nachricht --------
Betreff: Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]
Datum: Wed, 26 Oct 2011 22:38:47 +0200
Von: Martin Mailand<[email protected]>
Antwort an: [email protected]
An: Sage Weil<[email protected]>
Kopie (CC): Christian Brunner<[email protected]>, [email protected],
[email protected]

Hi,
I have more or less the same setup as Christian and I suffer the same problems.
But as far as I can see the output of latencytop and perf differs form Christian one, both are attached.
I was wondering about the high latency from btrfs-submit.

Process btrfs-submit-0 (970) Total: 2123.5 msec

I have as well the high IO rate and high IO wait.

avg-cpu: %user %nice %system %iowait %steal %idle

            0.60    0.00    2.20   82.40    0.00   14.80

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util

sda               0.00     0.00    0.00    8.40     0.00    74.40
17.71     0.03    3.81    0.00    3.81   3.81   3.20
sdb               0.00     7.00    0.00  269.80     0.00  1224.80

9.08 107.19 398.69 0.00 398.69 3.15 85.00

top - 21:57:41 up 8:41, 1 user, load average: 0.65, 0.79, 0.76
Tasks: 179 total, 1 running, 178 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.6%us, 2.4%sy, 0.0%ni, 70.8%id, 25.8%wa, 0.0%hi, 0.3%si, 0.0%st

Mem:   4018276k total,  1577728k used,  2440548k free,    10496k buffers
Swap:  1998844k total,        0k used,  1998844k free,  1316696k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1399 root 20 0 548m 103m 3428 S 0.0 2.6 2:01.85 ceph-osd

1401 root 20 0 548m 103m 3428 S 0.0 2.6 1:51.71 ceph-osd

1400 root 20 0 548m 103m 3428 S 0.0 2.6 1:50.30 ceph-osd

1391 root 20 0 0 0 0 S 0.0 0.0 1:18.39
btrfs-endio-wri

   976 root      20   0     0    0    0 S  0.0  0.0   1:18.11

btrfs-endio-wri

1367 root 20 0 0 0 0 S 0.0 0.0 1:05.60
btrfs-worker-1

   968 root      20   0     0    0    0 S  0.0  0.0   1:05.45

btrfs-worker-0

1163 root 20 0 141m 1636 1100 S 0.0 0.0 1:00.56 collectd

   970 root      20   0     0    0    0 S  0.0  0.0   0:47.73

btrfs-submit-0

1402 root 20 0 548m 103m 3428 S 0.0 2.6 0:34.86 ceph-osd

1392 root 20 0 0 0 0 S 0.0 0.0 0:33.70
btrfs-endio-met

   975 root      20   0     0    0    0 S  0.0  0.0   0:32.70

btrfs-endio-met

1415 root 20 0 548m 103m 3428 S 0.0 2.6 0:28.29 ceph-osd

1414 root 20 0 548m 103m 3428 S 0.0 2.6 0:28.24 ceph-osd

1397 root 20 0 548m 103m 3428 S 0.0 2.6 0:24.60 ceph-osd

1436 root 20 0 548m 103m 3428 S 0.0 2.6 0:13.31 ceph-osd

Here ist my setup.
Kernel v3.1 + Josef

The config for this osd (ceph version 0.37 (commit:a6f3bbb744a6faea95ae48317f0b838edb16a896)) is:
[osd.1]
host = s-brick-003
osd journal = /dev/sda7
btrfs devs = /dev/sdb
btrfs options = noatime filestore_btrfs_snap = false

I hope this helps to pin point the problem.

Best Regards,
martin

Quoted 56 lines · Sage Weil

Quoted 33 lines · Christian Brunner

2011/10/26 Sage Weil<[email protected]>:

Quoted 31 lines · Christian Brunner

Quoted 20 lines

Quoted 20 lines

Quoted 19 lines

Quoted 8 lines

Christian, have you tweaked those settings in your ceph.conf? It would be
something like 'journal dio = false'. If not, can you verify that directio shows true when the journal is initialized from your osd log?
E.g.,

Quoted 5 lines

  2011-10-21 15:21:02.026789 7ff7e5c54720 journal _open

dev/osd0.journal fd 14: 104857600 bytes, block size 4096 bytes, directio = 1

If directio = 1 for you, something else funky is causing those blkdev_fsync's...

I've looked it up in the logs - directio is 1:

Oct 25 17:20:16 os00 osd.000[1696]: 7f0016841740 journal _open

/dev/vg01/lv_osd_journal_0 fd 15: 17179869184 bytes, block size 4096 bytes, directio = 1

Quoted 5 lines

Quoted 3 lines

Do you mind capturing an strace? I'd like to see where that blkdev_fsync
is coming from.

Here is an strace. I can see a lot of sync_file_range operations.

Yeah, these all look like the flusher thread, and shouldn't be hitting blkdev_fsync. Can you confirm that with

Quoted 7 lines

Quoted 5 lines

filestore flusher = false
filestore sync flush = false

you get no sync_file_range at all? I wonder if this is also perf lying about the call chain.

Yes, setting this makes the sync_file_range calls go away.

Okay. That means either sync_file_range on a regular btrfs file is triggering blkdev_fsync somewhere in btrfs, there is an extremely sneaky bug that is mixing up file descriptors, or latencytop is lying. I'm guessing the latter, given the other weirdness Josef and Chris were seeing. :)

Quoted 1 line

Is it safe to use these settings with "filestore btrfs snap = 0"?

Yeah. They're purely a performance thing to push as much dirty data to disk as quickly as possible to minimize the snapshot create latency. You'll notice the write throughput tends to tank when them off.

sage