Is anyone using btrfs send/receive for backups instead of rsync?

Currently, I have a nightly cronjob that rsyncs my SSD to my HD on my laptop.

Of course, running rsync on an entire FS every night is slow, especially if I happen to try to use the laptop at the same time.

Obviously, that's what btrfs send/receive is supposed to fix:
https://btrfs.wiki.kernel.org/index.php/Design_notes_on_Send/Receive
http://lwn.net/Articles/506244/

It's been around for about 1.5 years now, so I was curious if others had converted to it for backups when indeed the wiki has the CYA 'if you use for backups, you get what you deserve' :)

Is it still being worked on, or kind of frozen/good as is?

There are also silent content missmatches on some subset of files that are not linearly written (vm images, databases and such).

https://bugzilla.kernel.org/show_bug.cgi?id=66941

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Emil, your bug report mentions the subvolume journal is different, and not anything about silent content differences on file data (images, databases, etc.) as you mention above. How are they related?

Also, how is the journal different on the receive-side? Is it "just" a size/allocation issue, as David commented on the bug report; or is the content actually different? For example, can it still be used to recover the filesystem post-crash? What does "fixed in V2" mean:
BTRFSv2 ?

Quoted 3 lines

Emil, your bug report mentions the subvolume journal is different, and not anything about silent content differences on file data (images, databases, etc.) as you mention above. How are they related?

systemd-journald is a logging daemon in the userspace that has nothing to do with filesystem journal.

Quoted 6 lines

Also, how is the journal different on the receive-side? Is it "just" a size/allocation issue, as David commented on the bug report; or is the content actually different? For example, can it still be used to recover the filesystem post-crash? What does "fixed in V2" mean:
BTRFSv2 ?

Send stream format specification V2.

Is the journal on the received side useless?

In my case, I'm unable to reproduce the problem. I get the same sha256sum of an existing 16M journal file (in the read only snapshot) as I do in the send/received snapshot. The difference is the original is compromised of 1976 extents, while the received is in a single extent.

Quoted 1 line

In my case, I'm unable to reproduce the problem. I get the same sha256sum of an existing 16M journal file (in the read only snapshot) as I do in the send/received snapshot. The difference is the original is compromised of 1976 extents, while the received is in a single extent.

Was the send incremental?

I am slightly bugged about a 16MB file having nearly 2000 extents, basically it's being turned into a bunch of 8KB fragments. I know nothing of the pros and cons of how systemd is writing journals, but they don't seem very big so I don't understand why they're preallocated, which on btrfs appears instantly defeated by COW upon the journal being modified. It seems to me either the journal doesn't need to be preallocated (at least on btrfs) or maybe systemd should set xattr +C on /var/log/journal?

That does disable checksumming though, along with data cow.

Yes, it does turn off checksumming as well as COW, but given the write- into scenario, that's actually best anyway, because otherwise btrfs has to keep updating the checksums

But in all these cases, it's also quite common for the application doing the writing to have its own checksumming/error-detection and possible correction -- it pretty much comes with the territory -- in which case btrfs attempting to do the same is simply superfluous even if it weren't a race-condition trigger.

And I'm predicting that since btrfs is the assumed successor to the ext* series as the Linux default filesystem, and systemd is targeting Linux default initsystem status as well, it's only logical that at some point systemd will detect what filesystem it's logging to, and will automatically set NOCOW on the journal file when that filesystem is btrfs.

Quoted 7 lines

And I'm predicting that since btrfs is the assumed successor to the ext*
series as the Linux default filesystem, and systemd is targeting Linux default initsystem status as well, it's only logical that at some point systemd will detect what filesystem it's logging to, and will automatically set NOCOW on the journal file when that filesystem is btrfs.

Is this something that should be brought up on the systemd-devel@ list? Or maybe file it as an RFE against systemd at freedesktop.org?

Quoted 3 lines · Duncan

Yes, it does turn off checksumming as well as COW, but given the write- into scenario, that's actually best anyway, because otherwise btrfs has to keep updating the checksums

On second thought, I'm less concerned with bitrot and checksumming being lost with nodatacow, than I am with significantly increasing the chance the journal is irreparably lost due to corruption during an unclean shutdown.

Quoted 6 lines

But in all these cases, it's also quite common for the application doing the writing to have its own checksumming/error-detection and possible correction -- it pretty much comes with the territory -- in which case btrfs attempting to do the same is simply superfluous even if it weren't a race-condition trigger.

I don't know what kind of checksumming systemd performs on the journal, but whenever Btrfs has found corruption with the journal file(s), systemd-journald has also found corruption and starts a new log. So it makes sense to rely on its own mechanisms, than Btrfs's.

Quoted 1 line

Is it still being worked on, or kind of frozen/good as is?

It's almost feature-complete: there's one version bump expected in
the stream format, to handle files with holes in. There are probably still awkward bugs in there (awkward in the sense that your backups will fail noisily if you hit one), but for my current usage -- disk to disk on the same system -- it seems to work OK.

I'm trying at the moment to hack together a working send/receive
backup between two systems. As far as I know, the send/receive will work OK, but I'm having problems with firewalls and ssh tunnels right now. :)

For instance, if I use an existing rsync destination to start syncing btrfs snapshots to after that, and one file operation can't be applied because let's say the destination file it's supposed to be applied to, isn't there?

Quoted 4 lines · Marc MERLIN

For instance, if I use an existing rsync destination to start syncing btrfs snapshots to after that, and one file operation can't be applied because let's say the destination file it's supposed to be applied to, isn't there?

I should have written more: I'm guessing what happens is that the btrfs receive fails/aborts, I get an error, I then run a manual rsync to reset everything to a good known state, and then continue the btrfs send/receive after that?

Quoted 6 lines · Marc MERLIN

Quoted 4 lines · Marc MERLIN

For instance, if I use an existing rsync destination to start syncing btrfs snapshots to after that, and one file operation can't be applied because let's say the destination file it's supposed to be applied to, isn't there?

Quoted 4 lines

I should have written more: I'm guessing what happens is that the btrfs receive fails/aborts, I get an error, I then run a manual rsync to reset everything to a good known state, and then continue the btrfs send/receive after that?

Btrfs send/receive works by matching state between snapshots on the sending and receiving end. If you update the files manually on the receiving end (say with rsync), it can't merge the states anymore.

Quoted 7 lines · Chris Mason

Quoted 6 lines · Marc MERLIN

Quoted 4 lines · Marc MERLIN

For instance, if I use an existing rsync destination to start syncing btrfs snapshots to after that, and one file operation can't be applied because let's say the destination file it's supposed to be applied to, isn't there?

Quoted 8 lines

Quoted 4 lines

I should have written more: I'm guessing what happens is that the btrfs receive fails/aborts, I get an error, I then run a manual rsync to reset everything to a good known state, and then continue the btrfs send/receive after that?

Btrfs send/receive works by matching state between snapshots on the sending and receiving end. If you update the files manually on the receiving end (say with rsync), it can't merge the states anymore.

I got that, but it wasn't quite my question :)

I understand that btrfs receive cannot apply file changes if the destination filesystem isn't in a file state that's identical to the source one.

I'm just not too sure how the destination FS needs to be configured so that btrfs receive can work with it.

1) Does it need to be an exact byte for byte copy of the block device the source was on?

2) Or can the destination be seeded with a full rsync or cp -a and can btrfs receive take over from there?

3) Then, if I hit a bug where something doesn't get synced right, and I run rsync to fix or verify that the two FS are indeed identical file-wise like they're supposed to, if rsync fixes something, are you saying that it'll stop btrfs receive from working after that?

Quoted 3 lines

1) Does it need to be an exact byte for byte copy of the block device the source was on?

No, in fact this doesn't help.

Quoted 2 lines

2) Or can the destination be seeded with a full rsync or cp -a and can btrfs receive take over from there?

No, it has to be created by btrfs receive.

Quoted 4 lines

3) Then, if I hit a bug where something doesn't get synced right, and I run rsync to fix or verify that the two FS are indeed identical file-wise like they're supposed to, if rsync fixes something, are you saying that it'll stop btrfs receive from working after that?

Yes, today anyway it won't work. Send converts the changed items into an intermediate format (we don't send btree blocks directly over the wire) and then receive modifies the destination from userland.

At the end of the stream we update the destination root to say "you're now version xxyyzz of uuid aabbcc".

We definitely could add a way to manually set this, but once a user does it, it'll be very hard to debug any problems they might have had if their copy wasn't actually up to date.

If one day, it could at least work on a subvolume level (only sync a subvolume), then it would be more useful to me. Maybe later…

Quoted 3 lines · Marc MERLIN

If one day, it could at least work on a subvolume level (only sync a subvolume), then it would be more useful to me. Maybe later…

Maybe I'm missing something, but btrfs send/receive only work on a subvolume level.

Quoted 6 lines · Chris Murphy

Quoted 3 lines · Marc MERLIN

If one day, it could at least work on a subvolume level (only sync a subvolume), then it would be more useful to me. Maybe later…

Quoted 1 line

Maybe I'm missing something, but btrfs send/receive only work on a subvolume level.

Never mind, I seem to be the one being dense. I mis-read that you needed to create the filesystem with btrfs receive.
Indeed, it's on a subvolume level, so it's actually fine since it does allow over provisionning afterall.

Quoted 6 lines · Chris Murphy

Quoted 3 lines · Marc MERLIN

If one day, it could at least work on a subvolume level (only sync a subvolume), then it would be more useful to me. Maybe later…

Quoted 1 line

Maybe I'm missing something, but btrfs send/receive only work on a subvolume level.

Never mind, I seem to be the one being dense. I mis-read that you needed to create the filesystem with btrfs receive.
Indeed, it's on a subvolume level, so it's actually fine since it does allow over provisionning afterall.

Quoted 7 lines · Marc MERLIN

Quoted 6 lines · Chris Murphy

Quoted 3 lines · Marc MERLIN

If one day, it could at least work on a subvolume level (only sync a subvolume), then it would be more useful to me. Maybe later…

Quoted 6 lines

Quoted 1 line

Maybe I'm missing something, but btrfs send/receive only work on a subvolume level.

Never mind, I seem to be the one being dense. I mis-read that you needed to create the filesystem with btrfs receive.
Indeed, it's on a subvolume level, so it's actually fine since it does allow over provisionning afterall.

Mmmh, but I just realized that on my laptop, I do boot the btrfs copy (currently done with rsync) from time to time (i.e. emergency boot from the HD the SSD was copied to).
If I do that, it'll change the filesystem that was created with btrfs receive and break it, preventing further updates, correct?

If so, can I get around that by making a boot snapshot after each copy and mount that snapshot for emergency boot instead of the main volume?

I read different howtos on the wiki and oracle docs, but I can't get it to work:

legolas:/mnt/btrfs_pool1# btrfs subvolume snapshot -r tmp tmp_read_only_new
Create a readonly snapshot of 'tmp' in './tmp_read_only_new'
legolas:/mnt/btrfs_pool1# sync
legolas:/mnt/btrfs_pool1# btrfs send tmp_read_only_new | btrfs receive /mnt/btrfs_pool2/
At subvol tmp_read_only_new
At subvol tmp_read_only_new

# Make a new snapshot later and try to sync it:

legolas:/mnt/btrfs_pool1# mv tmp_read_only_new tmp_read_only
legolas:/mnt/btrfs_pool1# btrfs subvolume snapshot -r tmp tmp_read_only_new
Create a readonly snapshot of 'tmp' in './tmp_read_only_new'
legolas:/mnt/btrfs_pool1# btrfs send -p tmp_read_only  tmp_read_only_new | btrfs receive /mnt/btrfs_pool2/
At subvol tmp_read_only_new
At snapshot tmp_read_only_new
ERROR: creating snapshot tmp_read_only_new -> tmp_read_only_new failed. File exists

This is where I get stuck:
Obviously /mnt/btrfs_pool2/tmp_read_only_new already exists since it's the reference backup volume.

What am I doing wrong?

You need to move /mnt/btrfs_pool2/tmp_read_only_new to a different
name as well. The send stream contains the name of the subvolume it wants to create, so it's trying to create a subvolume called "tmp_read_only_new" in /mnt/btrfs_pool2, and there's one there already.

Quoted 4 lines · Hugo Mills

You need to move /mnt/btrfs_pool2/tmp_read_only_new to a different
name as well. The send stream contains the name of the subvolume it wants to create, so it's trying to create a subvolume called "tmp_read_only_new" in /mnt/btrfs_pool2, and there's one there already.

Aaah, got it, thanks.

I'll contribute back a shell script that makes this much easier when I'm done with it.

Quoted 9 lines · Marc MERLIN

Quoted 4 lines · Hugo Mills

You need to move /mnt/btrfs_pool2/tmp_read_only_new to a different
name as well. The send stream contains the name of the subvolume it wants to create, so it's trying to create a subvolume called "tmp_read_only_new" in /mnt/btrfs_pool2, and there's one there already.

Aaah, got it, thanks.

I'll contribute back a shell script that makes this much easier when I'm done with it.

I've now spent enough time on this and have enough testing to release it.

http://marc.merlins.org/perso/btrfs/post_2014-03-22_Btrfs-Tips_-Doing-Fast-Incremental-Backups-With-Btrfs-Send-and-Receive.html
which points to
http://marc.merlins.org/linux/scripts/btrfs-subvolume-backup

Just for the archives, here is a working copy (the blog post above explains why it keeps multiple snapshots after the fact, and creates writeable ones on the destination, but basically this allows you to automatically boot from the latest snapshot (a writeable snapshot copy pointed to by a symlink tha tis kept up to date)):
----------------------------------------------------------------------------
#!/bin/bash

# By Marc MERLIN <[email protected]>
# License: GPL-2 or BSD at your choice.

# Source: http://marc.merlins.org/linux/scripts/
# $Id: btrfs-subvolume-backup 958 2014-03-16 00:23:28Z svnuser $

# cron jobs might not have /sbin in their path.
export PATH="$PATH:/sbin"

set -o nounset
set -o errexit
set -o pipefail

# From https://btrfs.wiki.kernel.org/index.php/Incremental_Backup

# bash shortcut for `basename $0`
PROG=${0##*/}
lock=/var/run/$PROG

usage() {
cat <<EOF
Usage:
cd /mnt/source_btrfs_pool
$PROG [--init] [--keep|-k num] [--dest hostname] volume_name /mnt/backup_btrfs_pool

Options:
--init: Print this help message and exit.
--keep num: Keep the last snapshots for local backups (5 by default)
--dest hostname: If present, ssh to that machine to make the copy.

This will snapshot volume_name in a btrfs pool, and send the diff between it and the previous snapshot (volume_name.last) to another btrfs pool (on other drives)

If your backup destination is another machine, you'll need to add a few ssh commands this script

The num sanpshots to keep is to give snapshots you can recover data from and they get deleted after num runs. Set to 0 to disable (one snapshot will be kept since it's required for the next diff to be computed). EOF
exit 0
}

die () {
     msg=${1:-}
     # don't loop on ERR
     trap '' ERR

     rm $lock

     echo "$msg" >&2
     echo >&2

     # This is a fancy shell core dumper
     if echo $msg | grep -q 'Error line .* with status'; then
	line=`echo $msg | sed 's/.*Error line \(.*\) with status.*/\1/'`
	echo " DIE: Code dump:" >&2
	nl -ba $0 | grep -3 "\b$line\b" >&2
     fi
     
     exit 1
}

# Trap errors for logging before we die (so that they can be picked up

# by the log checker)
trap 'die "Error line $LINENO with status $?"' ERR

init=""
# Keep the last 5 snapshots by default
keep=5
TEMP=$(getopt --longoptions help,usage,init,keep:,dest: -o h,k:,d: -- "$@") || usage dest=localhost
ssh=""

# getopt quotes arguments with ' We use eval to get rid of that
eval set -- $TEMP

while :
do

     case "$1" in
         -h|--help|--usage)
             usage
             shift
             ;;

	--keep|-k)
	    shift
	    keep=$1
	    shift
	    ;;

	--dest|-d)
	    shift
	    dest=$1
	    ssh="ssh $dest"
	    shift
	    ;;

	--init)
	    init=1
	    shift
	    ;;

	--)
	    shift
	    break
	    ;;

         *)
	    echo "Internal error!"
	    exit 1
	    ;;
     esac

done
[[ $keep < 1 ]] && die "Must keep at least one snapshot for things to work ($keep given)"

DATE="$(date '+%Y%m%d_%H:%M:%S')"

[[ $# != 2 ]] && usage
vol="$1"
dest_pool="$2"

# shlock (from inn) does the right thing and grabs a lock for a dead process
# (it checks the PID in the lock file and if it's not there, it
# updates the PID with the value given to -p)
if ! shlock -p $$ -f $lock; then
     echo "$lock held for $PROG, quitting" >&2
     exit
fi

if [[ -z "$init" ]]; then

     test -e "${vol}_last" \
	|| die "Cannot sync $vol, ${vol}_last missing. Try --init?"
     src_snap="$(readlink -e ${vol}_last)"

fi
src_newsnap="${vol}_ro.$DATE"
src_newsnaprw="${vol}_rw.$DATE"

$ssh test -d "$dest_pool/" || die "ABORT: $dest_pool not a directory (on $dest)"

btrfs subvolume snapshot -r "$vol" "$src_newsnap"

# There is currently an issue that the snapshots to be used with "btrfs send"

# must be physically on the disk, or you may receive a "stale NFS file handle"
# error. This is accomplished by "sync" after the snapshot
sync

if [[ -n "$init" ]]; then
btrfs send "$src_newsnap" | $ssh btrfs receive "$dest_pool/"
else
btrfs send -p "$src_snap" "$src_newsnap" | $ssh btrfs receive "$dest_pool/"
fi

# We make a read-write snapshot in case you want to use it for a chroot

# and some testing with a writeable filesystem or want to boot from a
# last good known snapshot.
btrfs subvolume snapshot "$src_newsnap" "$src_newsnaprw"
$ssh btrfs subvolume snapshot "$dest_pool/$src_newsnap" "$dest_pool/$src_newsnaprw"

# Keep track of the last snapshot to send a diff against. ln -snf $src_newsnap ${vol}_last
# The rw version can be used for mounting with subvol=vol_last_rw ln -snf $src_newsnaprw ${vol}_last_rw
$ssh ln -snf $src_newsnaprw $dest_pool/${vol}_last_rw

# How many snapshots to keep on the source btrfs pool (both read

# only and read-write).
ls -rd ${vol}_ro* | tail -n +$(( $keep + 1 ))| while read snap
do
     btrfs subvolume delete "$snap"
done
ls -rd ${vol}_rw* | tail -n +$(( $keep + 1 ))| while read snap
do
     btrfs subvolume delete "$snap"
done

# Same thing for destination (assume the same number of snapshots to keep,

# you can change this if you really want).
$ssh ls -rd $dest_pool/${vol}_ro* | tail -n +$(( $keep + 1 ))| while read snap
do
     $ssh btrfs subvolume delete "$snap"
done
$ssh ls -rd $dest_pool/${vol}_rw* | tail -n +$(( $keep + 1 ))| while read snap
do
     $ssh btrfs subvolume delete "$snap"
done

rm $lock
----------------------------------------------------------------------------

Quoted 1 line · Marc MERLIN

Hi, Marc

Feel free to use ideas from my own script. Some aspects in my script are more mature and others are frankly pathetic. ;)

There are also quite a lot of TODOs throughout my script that aren't likely to get the urgent attention they deserve. It has been slowly evolving over the last two weeks.

http://swiftspirit.co.za/scripts/btrfs-snd-rcv-backup

Hi, Marc

Feel free to use ideas from my own script. Some aspects in my script are more mature and others are frankly pathetic. ;)

There are also quite a lot of TODOs throughout my script that aren't likely to get the urgent attention they deserve. It has been slowly evolving over the last two weeks.

http://swiftspirit.co.za/scripts/btrfs-snd-rcv-backup

From a quick read, it looks even more complex than mine :) but

Quoted 5 lines · Brendan Hide

Hi, Marc

Feel free to use ideas from my own script. Some aspects in my script are more mature and others are frankly pathetic. ;)

Quoted 5 lines

There are also quite a lot of TODOs throughout my script that aren't likely to get the urgent attention they deserve. It has been slowly evolving over the last two weeks.

http://swiftspirit.co.za/scripts/btrfs-snd-rcv-backup

I figured I likely wasn't the only one working on a script like this :)

From a quick read, it looks even more complex than mine :) but

- it doesn't do ssh to a destination for a remote backup

- it doesn't seem to keep a list of configurable snapshots not necessary for send/restore but useful for getting historical data

- it doesn't seem to use a symlink to keep track of the last complete snapshot on the source and destination, and does more work to compensate when recovering from an incomplete backup/restore.

- it doesn't create writeable snapshots on the destination in case you want to use the copy as a live filesystem

Things I noticed:
- I don't use ionice, maybe I should. Did you find that it actually made a difference with send/receive?

- Your comments say shlock isn't safe and that's documented. I don't see that in the man page
http://manpages.ubuntu.com/manpages/trusty/man1/shlock.1.html

I'd love to have details on this if I shouldn't be using it
- Is set -o noclobber; echo $$ > $lockfile really atomic and safer than shlock? If so, great, although I would then wonder why shlock even exists :)

Thanks,
Marc

Quoted 2 lines

- it doesn't create writeable snapshots on the destination in case you want to use the copy as a live filesystem

One of the issues with doing writeable snapshots by default is that the backup is (ever so slightly) less safe from "fat-finger syndrome". If I want a writeable snapshot, I'll make it from the read-only snapshot, thereby reducing the chances of accidentally "tainting" or deleting data in the backup. I actually *did* accidentally delete my entire filesystem (hence the paranoid umounts). But, of course, my script *first* created read-only snapshots from which recovery took only a few minutes. ;)

Quoted 3 lines

Things I noticed:
- I don't use ionice, maybe I should. Did you find that it actually made a difference with send/receive?

This is just a habit I've developed over time in all my scripts. I figure that if I'm using the machine at the time and the snapshot has a large churn, I'd prefer the ionice. That said, the main test system is a desktop which is likely to have much less churn than a server. In the last two weeks the longest daily incremental backup took about 5 minutes to complete, while it typically takes about 30 seconds only.

Quoted 3 lines

- Your comments say shlock isn't safe and that's documented. I don't see that in the man page
http://manpages.ubuntu.com/manpages/trusty/man1/shlock.1.html

That man page looks newer than the one I last looked at - specifically the part saying "improved by Berend Reitsma to solve a race condition." The previous documentation on shlock indicated it was safe for hourly crons - but not in the case where a cron might be executed twice simultaneously. Shlock was recommended by a colleague until I realised this potential issue, thus my template doesn't use it. I should update the comment with some updated information.

Quoted 4 lines

I'd love to have details on this if I shouldn't be using it
- Is set -o noclobber; echo $$ > $lockfile really atomic and safer than shlock? If so, great, although I would then wonder why shlock even exists :)

The part that brings about an atomic lock is "noclobber", which sets it so that we are not allowed to "clobber"/"overwrite" an existing file. Thus, if the file exists, the command fails. If it successfully creates the new file, the command returns true.

I'd consider changing this mostly for the fact that depending on INN is a very big dependency. There are other options as well, though I don't think they're as portable as noclobber.

Please consider adding a blank line between quotes, it makes them just a bit more readable :)

Quoted 9 lines

Quoted 2 lines

- it doesn't create writeable snapshots on the destination in case you want to use the copy as a live filesystem

One of the issues with doing writeable snapshots by default is that the backup is (ever so slightly) less safe from "fat-finger syndrome". If I want a writeable snapshot, I'll make it from the read-only snapshot, thereby reducing the chances of accidentally "tainting" or deleting data in the backup. I actually *did* accidentally delete my entire filesystem (hence the paranoid umounts). But, of course, my script *first* created read-only snapshots from which recovery took only a few minutes. ;)

The writeable snapshot I create is on top of the read only one used by btrfs receive. So, I can play with it, but it won't upset/break anything for the backup.

The historical snapshots I keep give me cheap backups to go back to do get a file I may have deleted 3 days ago and want back now even though my btrfs send/receive runs hourly.

[snip]

Quoted 10 lines

Quoted 3 lines

- Your comments say shlock isn't safe and that's documented. I don't see that in the man page
http://manpages.ubuntu.com/manpages/trusty/man1/shlock.1.html

That man page looks newer than the one I last looked at - specifically the part saying "improved by Berend Reitsma to solve a race condition." The previous documentation on shlock indicated it was safe for hourly crons - but not in the case where a cron might be executed twice simultaneously. Shlock was recommended by a colleague until I realised this potential issue, thus my template doesn't use it. I should update the comment with some updated information.

It's not super important, it was more my curiosity.
If a simple lock program in C isn't atomic, what's the point of it? I never looked at the source code, but maybe I should...

Quoted 4 lines

Quoted 4 lines

I'd love to have details on this if I shouldn't be using it
- Is set -o noclobber; echo $$ > $lockfile really atomic and safer than shlock? If so, great, although I would then wonder why shlock even exists :)

Quoted 4 lines

The part that brings about an atomic lock is "noclobber", which sets it so that we are not allowed to "clobber"/"overwrite" an existing file. Thus, if the file exists, the command fails. If it successfully creates the new file, the command returns true.

I understand how it's supposed to work, I just wondered if it was really atomic as it should be since there would be no reason for shlock to even exist with that line of code you wrote.

Thanks for the info Marc