Skip to contentSkip to conversation

Sync semantics.

DISCUSSION Rogier Wolff 4 participants 7 messages Nov 11, 2010 – Nov 17, 2010

lore
Share & tools
lore
#0 lore raw

Hi,

What should I expect from a "sync" system call?

The manual says:

sync() first commits inodes to buffers, and then buffers to disk.

and then goes on to state:

... since version 1.3.20 Linux does actually wait.

[for the buffers to be handed over to the drive]

So how long can I expect a "sync" call to take?

I would expect that all buffers that are dirty at the time of the "sync" call are written by the time that sync returns. I'm currently bombarding my fileserver with some 40-60Mbytes per second of data to be written (*). The fileserver has 8G of memory. So max 8000 Mb of dirty buffers can be stored, right? The server writes an average of (at least) 40Mb/second to disk. According to my calculator, I will have to wait up to 200 seconds for the sync system call to return....

# time sync
0.000u 0.220s 2:22:23.96 0.0%   0+0k 0+0io 2pf+0w

Two hours 22 minutes.

I typed the "time sync" again, and it hasn't returned yet.... Actually I don't expect it to before 6-hours-from-now because that's when the clients will run out of data to send.

wolff    13706  0.0  0.0   1816   208 pts/12   D+   12:08   0:00 sync
wolff    14116  0.0  0.0   1908   520 pts/34   S+   13:48   0:00 grep sync

It's been running 100 minutes by now.....

(*) The three clients are each sending 20-35 Mb/second but are being held up by the server who doesn't seem to be handling more than about 40-50Mb/sec total.

Signature · 6 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
#1 lore raw
Quoted 4 lines · Rogier Wolff

What should I expect from a "sync" system call?

# time sync
0.000u 0.220s 2:22:23.96 0.0%   0+0k 0+0io 2pf+0w
Two hours 22 minutes. 

I would also be interested to hear the answer for this question. Sync and ongoing i/o simply don't seem to play well together, and you don't even need anything as heavy as mentioned above.

Michal Svoboda

#2 lore raw
In reply to Rogier Wolff
Quoted 26 lines · Rogier Wolff

Hi,

What should I expect from a "sync" system call?

The manual says:

sync() first commits inodes to buffers, and then buffers to disk.

and then goes on to state:

... since version 1.3.20 Linux does actually wait.

[for the buffers to be handed over to the drive]

So how long can I expect a "sync" call to take?

I would expect that all buffers that are dirty at the time of the "sync" call are written by the time that sync returns. I'm currently bombarding my fileserver with some 40-60Mbytes per second of data to be written (*). The fileserver has 8G of memory. So max 8000 Mb of dirty buffers can be stored, right? The server writes an average of (at least) 40Mb/second to disk. According to my calculator, I will have to wait up to 200 seconds for the sync system call to return....

Quoted 2 lines
# time sync
0.000u 0.220s 2:22:23.96 0.0%   0+0k 0+0io 2pf+0w

Depending on the kernel, sync will keep writing if you keep dirtying. This should be mostly fixed in 2.6.36....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#3 lore raw
Quoted 2 lines · Dave Chinner

Depending on the kernel, sync will keep writing if you keep dirtying. This should be mostly fixed in 2.6.36....

Is that a "we hope that it is so but we are not sure" kind of "mostly", or a "there are known cases when this is not true" one?

Michal Svoboda

#4 lore raw
Quoted 6 lines · Michal Svoboda
Quoted 2 lines · Dave Chinner

Depending on the kernel, sync will keep writing if you keep dirtying. This should be mostly fixed in 2.6.36....

Is that a "we hope that it is so but we are not sure" kind of "mostly", or a "there are known cases when this is not true" one?

The latter.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#5 lore raw
In reply to Rogier Wolff

Hi!

Quoted 10 lines

What should I expect from a "sync" system call?

The manual says:

sync() first commits inodes to buffers, and then buffers to disk.

and then goes on to state:

... since version 1.3.20 Linux does actually wait.

Quoted 8 lines

[for the buffers to be handed over to the drive]

So how long can I expect a "sync" call to take?

I would expect that all buffers that are dirty at the time of the "sync" call are written by the time that sync returns. I'm currently bombarding my fileserver with some 40-60Mbytes per second of data to be written (*). The fileserver has 8G of memory. So max 8000 Mb of

Are you sure? Hitting 40MB/sec is hard when it involves seeking...

You may want to lower dirty_ratio...

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#6 lore raw
Quoted 6 lines · Pavel Machek
Quoted 4 lines

I would expect that all buffers that are dirty at the time of the "sync" call are written by the time that sync returns. I'm currently bombarding my fileserver with some 40-60Mbytes per second of data to be written (*). The fileserver has 8G of memory. So max 8000 Mb of

Are you sure? Hitting 40MB/sec is hard when it involves seeking...

Yeah... It's about 10 times slower than when no seeking is involved, so that makes sense, doesn't it? The machine can sustain over 400 Mb per second on linear reads:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  0      0  50908 6667292 502040    0    0 430064     0 2171 1677  0 23 66 11
 4  0      0  51280 6713952 501976    0    0 429596     0 2430 1889 16 28 44 12
 1  0      0  51768 6754884 502100    0    0 423388     0 2460 2100 13 28 47 13
 0  1      0  50760 6793392 502416    0    0 422892     0 2174 1796  0 21 68 10

Through the filesystem I get:

1073741824 bytes (1.1 GB) copied, 2.70151 s, 397 MB/s 1073741824 bytes (1.1 GB) copied, 2.62782 s, 409 MB/s

Which impresses me. In practise I seldomly see high 1xxMb/sec. (i.e. 120-150Mb per second happens, while 180-190 is rare).

On the other hand, in the same run I also get:
1073741824 bytes (1.1 GB) copied, 6.82678 s, 157 MB/s 1073741824 bytes (1.1 GB) copied, 6.66133 s, 161 MB/s 1073741824 bytes (1.1 GB) copied, 6.58995 s, 163 MB/s

which apparently is caused by these files being more fragmented. These files (1Gb each) were written linearly, but some might have been written wile other of these 1G files (in a different directory) were written at the same time. I'm guessing these ended up more or less interleaved.

Checking up on the fragmentation of these files, the fast ones have about 600-800 fragments, while the slow ones have 1300-2000 fragments.

Mb/sec #frags
400 1252

 493        865
 391        755
 393        606
 395        819
 206        937
 159        901
 173       1940
 165       1806
 157       1481
 168       1351
 179       2692
 166       1541
 154       1151
 159        924
 149       1228
 155       1139
 151       1103
 150       1070
 155       1160

There is SOME correlation but not 100%. This is on an 8x1T RAID.

Quoted 1 line

You may want to lower dirty_ratio...

You know, what I would REALLY want is that when say 400Mb of dirty buffers exist, the machine would start alternating between the two or three areas that require writing. All these should be "linear". If you switch only once every second or so, the "seeking time" is less than 1%. In that case, my server should be able to write up to 400Mb per second, except for that I can only supply 120Mb per second over the Ethernet. But that would still be a 3x improvement over what the machine can handle now.

In theory these things should work even better if things like "dirty_ratio" are higher.

In the current situation, the "sync" call will return when the IO system falls to "idle". The chances of "nothing needing writing" increases as the amount of allowed buffers is lower. But the problem is that sync keeps on waiting for those new "dirty" buffers that have become dirty AFTER the start of the sync call.

Suppose we have a mail handling daemon that just recieved an Email from over the network. Instead of just saying: Ok, i'll take over from here, it prefers to write it to disk, and calls sync, so that should the power fail, the EMail is on permanent storage, and can be correctly handled.

This works just great, until someone manages to get the server to continue to get new dirty buffers, so that the sync takes over ten minutes, and the other sides MTA will time out.....

Anyway, someone told me that it's been fixed, and sync won't behave like this anymore.

Roger.

Quoted 4 lines
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Signature · 6 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ