Skip to contentSkip to conversation

kernel.bkbits.net off the air

DISCUSSION walt 7 participants 13 messages Nov 11, 2003 – Nov 14, 2003 2 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

walt

author
#2 lore raw
Quoted 6 lines · Andrea Arcangeli
Quoted 1 line · Andrea Arcangeli

The best way to fix this isn't to add locking to rsync, but to add two files inside or outside the tree, each one is a sequence number, so you fetch file1 first, then you rsync and you fetch file2, then you compare them. If they're the same, your rsync copy is coherent. It's the same locking we introduced with vgettimeofday...

How is this different from writing one file named LOCK while updating the tree?

I know this is a really basic question, but it also seems like a really old problem which must have been solved multiple times by now. Am I really wrong about this?

#3 lore raw
Quoted 11 lines · walt
Quoted 6 lines · Andrea Arcangeli
Quoted 1 line · Andrea Arcangeli

The best way to fix this isn't to add locking to rsync, but to add two files inside or outside the tree, each one is a sequence number, so you fetch file1 first, then you rsync and you fetch file2, then you compare them. If they're the same, your rsync copy is coherent. It's the same locking we introduced with vgettimeofday...

How is this different from writing one file named LOCK while updating the tree?

This is even simpler I believe. If you happen to fetch it, you restart the rsync. Peter ?
(maybe the name LOCK should be replaced by something more "uniq")

- Davide

#4 lore raw
Quoted 14 lines · Davide Libenzi
Quoted 11 lines · walt
Quoted 8 lines · Andrea Arcangeli
Quoted 2 lines · Andrea Arcangeli

The best way to fix this isn't to add locking to rsync, but to add two files inside or outside the tree, each one is a sequence number, so you fetch file1 first, then you rsync and you fetch file2, then you compare them. If they're the same, your rsync copy is coherent. It's the same locking we introduced with vgettimeofday...

Quoted 8 lines
Quoted 3 lines

How is this different from writing one file named LOCK while updating the tree?

This is even simpler I believe. If you happen to fetch it, you restart the rsync. Peter ? (maybe the name LOCK should be replaced by something more "uniq")

What happens if the the tree is updated while the client is fetching it?

#5 lore raw
Quoted 1 line · Nick Piggin

What happens if the the tree is updated while the client is fetching it?

Surprise, it breaks :-) Yes, the double file approach is needed (excluding changes to rsync).

- Davide

#6 lore raw
In reply to Nick Piggin
Quoted 22 lines · Nick Piggin
Quoted 18 lines · Davide Libenzi
Quoted 14 lines · walt
Quoted 8 lines · Andrea Arcangeli
Quoted 2 lines · Andrea Arcangeli

The best way to fix this isn't to add locking to rsync, but to add two files inside or outside the tree, each one is a sequence number, so you fetch file1 first, then you rsync and you fetch file2, then you compare them. If they're the same, your rsync copy is coherent. It's the same locking we introduced with vgettimeofday...

How is this different from writing one file named LOCK while updating the tree?

Quoted 7 lines
Quoted 5 lines

This is even simpler I believe. If you happen to fetch it, you restart the rsync. Peter ?
(maybe the name LOCK should be replaced by something more "uniq")

What happens if the the tree is updated while the client is fetching it?

the usual problem, and the reason we need a sequence number (increased before and after the repo update). A file lock not.

#7 lore raw
Quoted 2 lines · Andrea Arcangeli

the usual problem, and the reason we need a sequence number (increased before and after the repo update). A file lock not.

Or a file that contains md5sums of the other files in the tree. After the rsync, you recompute the md5sums file, and if it does not match, rsync again. As a bonus feature, the md5sums file can be pgp-signed.

-- Benoit

#8 lore raw
Quoted 8 lines · Benoit Poulot-Cazajous
Quoted 2 lines · Andrea Arcangeli

the usual problem, and the reason we need a sequence number (increased before and after the repo update). A file lock not.

Or a file that contains md5sums of the other files in the tree. After the rsync, you recompute the md5sums file, and if it does not match, rsync again. As a bonus feature, the md5sums file can be pgp-signed.

agreed, this would work too and it has the advantage of working with the mirrors too as far as the per-file updates are atomic (should always be the case). This has the only disavanage of forcing the client and the server to read all file contents (I normally don't rsync with -c). But if it simplifies life a lot, it's sure acceptable.

#9 lore raw
Quoted 6 lines · Andrea Arcangeli
Quoted 5 lines · Benoit Poulot-Cazajous
Quoted 2 lines · Andrea Arcangeli

the usual problem, and the reason we need a sequence number (increased before and after the repo update). A file lock not.

Quoted 8 lines
Quoted 3 lines

Or a file that contains md5sums of the other files in the tree. After the rsync, you recompute the md5sums file, and if it does not match, rsync again. As a bonus feature, the md5sums file can be pgp-signed.

agreed, this would work too and it has the advantage of working with the mirrors too as far as the per-file updates are atomic (should always be the case). This has the only disavanage of forcing the client and the server to read all file contents (I normally don't rsync with -c).

This is not necessary, you only need to recompute the md5sums of changed files.

Andreas.

Signature · 4 lines
Andreas Schwab, SuSE Labs, [email protected]
SuSE Linux AG, Deutschherrnstr. 15-19, D-90429 Nürnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
#10 lore raw
Quoted 2 lines · Andreas Schwab

This is not necessary, you only need to recompute the md5sums of changed files.

agreed, theoretically I could intercept the changed files through rsync output, it's more tricky but doable.

#11 lore raw
In reply to Andreas Schwab
Quoted 12 lines · Andreas Schwab
Quoted 10 lines · Andrea Arcangeli
Quoted 8 lines · Benoit Poulot-Cazajous
Quoted 2 lines · Andrea Arcangeli

the usual problem, and the reason we need a sequence number (increased before and after the repo update). A file lock not.

Or a file that contains md5sums of the other files in the tree. After the rsync, you recompute the md5sums file, and if it does not match, rsync again. As a bonus feature, the md5sums file can be pgp-signed.

Quoted 7 lines
Quoted 4 lines

agreed, this would work too and it has the advantage of working with the mirrors too as far as the per-file updates are atomic (should always be the case). This has the only disavanage of forcing the client and the server to read all file contents (I normally don't rsync with -c).

This is not necessary, you only need to recompute the md5sums of changed files.

If we had this approach we wouldn't have caught the torjan horse in the CVS tree. We checksum all the data, changed or not. Your approach pushes that duty onto the end users, and let's have a show of hands, how many of you md5sum every bit of data that you download from the net? (Please don't reply to that, it's a rhetorical question and I think the vast majority of the people already know the answer.)

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#12 lore raw
Quoted 3 lines · Larry McVoy

If we had this approach we wouldn't have caught the torjan horse in the CVS tree. We checksum all the data, changed or not. Your approach pushes that duty onto the end users, and let's have a show of hands,

the suggestion of the md5sum wasn't related to keeping the data secure/robust, we don't want to move the "robustness" duty onto the end users of course, we only want to know when we can break the rsync loop. The fact we'll do a further check possibly with a signature on the md5sums, is a bonus, but it's not meant to replace in any way the robusteness effort on the server side and we don't do it for robustness, we do it only for providing a means of coherency to the changesets in the tree.

#13 lore raw
In reply to Andreas Schwab
Quoted 12 lines · Andreas Schwab
Quoted 10 lines · Andrea Arcangeli
Quoted 8 lines · Benoit Poulot-Cazajous
Quoted 2 lines · Andrea Arcangeli

the usual problem, and the reason we need a sequence number (increased before and after the repo update). A file lock not.

Or a file that contains md5sums of the other files in the tree. After the rsync, you recompute the md5sums file, and if it does not match, rsync again. As a bonus feature, the md5sums file can be pgp-signed.

Quoted 7 lines
Quoted 4 lines

agreed, this would work too and it has the advantage of working with the mirrors too as far as the per-file updates are atomic (should always be the case). This has the only disavanage of forcing the client and the server to read all file contents (I normally don't rsync with -c).

This is not necessary, you only need to recompute the md5sums of changed files.

Well, if you really want to optimize (md5sum is fast, faster than a typical hd, and much faster than gcc), check the most recent file. If its md5sum does not match, rsync again.
There are many ways to optimize the md5sum of the whole tree, when a previous tree with a trusted md5sum file is available. "find -ls" can help there. But make sure not to over-optimize, as servers with wrong md5sum files will be DDOSed by their clients ;-)

-- Benoit

#14 lore raw
In reply to walt
Quoted 11 lines · walt
Quoted 6 lines · Andrea Arcangeli
Quoted 1 line · Andrea Arcangeli

The best way to fix this isn't to add locking to rsync, but to add two files inside or outside the tree, each one is a sequence number, so you fetch file1 first, then you rsync and you fetch file2, then you compare them. If they're the same, your rsync copy is coherent. It's the same locking we introduced with vgettimeofday...

How is this different from writing one file named LOCK while updating the tree?

it's very different, your file named LOCK can be created and deleted while rsync was running, and rsync will never notice. rsync is readonly, it'll never care about locks.