kernel.bkbits.net off the air
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines · Andrea Arcangeli
Quoted 1 line · Andrea Arcangeli
The best way to fix this isn't to add locking to rsync, but to add two files inside or outside the tree, each one is a sequence number, so you fetch file1 first, then you rsync and you fetch file2, then you compare them. If they're the same, your rsync copy is coherent. It's the same locking we introduced with vgettimeofday...
How is this different from writing one file named LOCK while updating the tree?
I know this is a really basic question, but it also seems like a really old problem which must have been solved multiple times by now. Am I really wrong about this?
Quoted 11 lines · walt
Quoted 6 lines · Andrea Arcangeli
Quoted 1 line · Andrea Arcangeli
The best way to fix this isn't to add locking to rsync, but to add two files inside or outside the tree, each one is a sequence number, so you fetch file1 first, then you rsync and you fetch file2, then you compare them. If they're the same, your rsync copy is coherent. It's the same locking we introduced with vgettimeofday...
How is this different from writing one file named LOCK while updating the tree?
This is even simpler I believe. If you happen to fetch it, you restart the rsync. Peter ?
(maybe the name LOCK should be replaced by something more "uniq")
- Davide
Quoted 14 lines · Davide Libenzi
Quoted 11 lines · walt
Quoted 8 lines · Andrea Arcangeli
Quoted 2 lines · Andrea Arcangeli
The best way to fix this isn't to add locking to rsync, but to add two files inside or outside the tree, each one is a sequence number, so you fetch file1 first, then you rsync and you fetch file2, then you compare them. If they're the same, your rsync copy is coherent. It's the same locking we introduced with vgettimeofday...
Quoted 8 lines
Quoted 3 lines
How is this different from writing one file named LOCK while updating the tree?
This is even simpler I believe. If you happen to fetch it, you restart the rsync. Peter ? (maybe the name LOCK should be replaced by something more "uniq")
What happens if the the tree is updated while the client is fetching it?
Quoted 1 line · Nick Piggin
What happens if the the tree is updated while the client is fetching it?
Surprise, it breaks :-) Yes, the double file approach is needed (excluding changes to rsync).
- Davide
Quoted 22 lines · Nick Piggin
Quoted 18 lines · Davide Libenzi
Quoted 14 lines · walt
Quoted 8 lines · Andrea Arcangeli
Quoted 2 lines · Andrea Arcangeli
The best way to fix this isn't to add locking to rsync, but to add two files inside or outside the tree, each one is a sequence number, so you fetch file1 first, then you rsync and you fetch file2, then you compare them. If they're the same, your rsync copy is coherent. It's the same locking we introduced with vgettimeofday...
How is this different from writing one file named LOCK while updating the tree?
Quoted 7 lines
Quoted 5 lines
This is even simpler I believe. If you happen to fetch it, you restart the rsync. Peter ?
(maybe the name LOCK should be replaced by something more "uniq")What happens if the the tree is updated while the client is fetching it?
the usual problem, and the reason we need a sequence number (increased before and after the repo update). A file lock not.
Quoted 2 lines · Andrea Arcangeli
the usual problem, and the reason we need a sequence number (increased before and after the repo update). A file lock not.
Or a file that contains md5sums of the other files in the tree. After the rsync, you recompute the md5sums file, and if it does not match, rsync again. As a bonus feature, the md5sums file can be pgp-signed.
-- Benoit
Quoted 8 lines · Benoit Poulot-Cazajous
Quoted 2 lines · Andrea Arcangeli
the usual problem, and the reason we need a sequence number (increased before and after the repo update). A file lock not.
Or a file that contains md5sums of the other files in the tree. After the rsync, you recompute the md5sums file, and if it does not match, rsync again. As a bonus feature, the md5sums file can be pgp-signed.
agreed, this would work too and it has the advantage of working with the mirrors too as far as the per-file updates are atomic (should always be the case). This has the only disavanage of forcing the client and the server to read all file contents (I normally don't rsync with -c). But if it simplifies life a lot, it's sure acceptable.
Quoted 6 lines · Andrea Arcangeli
Quoted 5 lines · Benoit Poulot-Cazajous
Quoted 2 lines · Andrea Arcangeli
the usual problem, and the reason we need a sequence number (increased before and after the repo update). A file lock not.
Quoted 8 lines
Quoted 3 lines
Or a file that contains md5sums of the other files in the tree. After the rsync, you recompute the md5sums file, and if it does not match, rsync again. As a bonus feature, the md5sums file can be pgp-signed.
agreed, this would work too and it has the advantage of working with the mirrors too as far as the per-file updates are atomic (should always be the case). This has the only disavanage of forcing the client and the server to read all file contents (I normally don't rsync with -c).
This is not necessary, you only need to recompute the md5sums of changed files.
Andreas.
Signature · 4 lines
Andreas Schwab, SuSE Labs, [email protected] SuSE Linux AG, Deutschherrnstr. 15-19, D-90429 Nürnberg Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."
Quoted 2 lines · Andreas Schwab
This is not necessary, you only need to recompute the md5sums of changed files.
agreed, theoretically I could intercept the changed files through rsync output, it's more tricky but doable.
Quoted 12 lines · Andreas Schwab
Quoted 10 lines · Andrea Arcangeli
Quoted 8 lines · Benoit Poulot-Cazajous
Quoted 2 lines · Andrea Arcangeli
the usual problem, and the reason we need a sequence number (increased before and after the repo update). A file lock not.
Or a file that contains md5sums of the other files in the tree. After the rsync, you recompute the md5sums file, and if it does not match, rsync again. As a bonus feature, the md5sums file can be pgp-signed.
Quoted 7 lines
Quoted 4 lines
agreed, this would work too and it has the advantage of working with the mirrors too as far as the per-file updates are atomic (should always be the case). This has the only disavanage of forcing the client and the server to read all file contents (I normally don't rsync with -c).
This is not necessary, you only need to recompute the md5sums of changed files.
If we had this approach we wouldn't have caught the torjan horse in the CVS tree. We checksum all the data, changed or not. Your approach pushes that duty onto the end users, and let's have a show of hands, how many of you md5sum every bit of data that you download from the net? (Please don't reply to that, it's a rhetorical question and I think the vast majority of the people already know the answer.)
Signature · 2 lines
--- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
Quoted 3 lines · Larry McVoy
If we had this approach we wouldn't have caught the torjan horse in the CVS tree. We checksum all the data, changed or not. Your approach pushes that duty onto the end users, and let's have a show of hands,
the suggestion of the md5sum wasn't related to keeping the data secure/robust, we don't want to move the "robustness" duty onto the end users of course, we only want to know when we can break the rsync loop. The fact we'll do a further check possibly with a signature on the md5sums, is a bonus, but it's not meant to replace in any way the robusteness effort on the server side and we don't do it for robustness, we do it only for providing a means of coherency to the changesets in the tree.
Quoted 12 lines · Andreas Schwab
Quoted 10 lines · Andrea Arcangeli
Quoted 8 lines · Benoit Poulot-Cazajous
Quoted 2 lines · Andrea Arcangeli
the usual problem, and the reason we need a sequence number (increased before and after the repo update). A file lock not.
Or a file that contains md5sums of the other files in the tree. After the rsync, you recompute the md5sums file, and if it does not match, rsync again. As a bonus feature, the md5sums file can be pgp-signed.
Quoted 7 lines
Quoted 4 lines
agreed, this would work too and it has the advantage of working with the mirrors too as far as the per-file updates are atomic (should always be the case). This has the only disavanage of forcing the client and the server to read all file contents (I normally don't rsync with -c).
This is not necessary, you only need to recompute the md5sums of changed files.
Well, if you really want to optimize (md5sum is fast, faster than a typical hd, and much faster than gcc), check the most recent file. If its md5sum does not match, rsync again.
There are many ways to optimize the md5sum of the whole tree, when a previous tree with a trusted md5sum file is available. "find -ls" can help there. But make sure not to over-optimize, as servers with wrong md5sum files will be DDOSed by their clients ;-)
-- Benoit
Quoted 11 lines · walt
Quoted 6 lines · Andrea Arcangeli
Quoted 1 line · Andrea Arcangeli
The best way to fix this isn't to add locking to rsync, but to add two files inside or outside the tree, each one is a sequence number, so you fetch file1 first, then you rsync and you fetch file2, then you compare them. If they're the same, your rsync copy is coherent. It's the same locking we introduced with vgettimeofday...
How is this different from writing one file named LOCK while updating the tree?
it's very different, your file named LOCK can be created and deleted while rsync was running, and rsync will never notice. rsync is readonly, it'll never care about locks.