Using make and rsync for Data Replication

When maintaining a cluster environment (such as HP Serviceguard) there are often directories and configurations which need to be maintained on two different local disks (on different machines). Using make and rsync (with ssh) are excellent for this.

The rsync command allows you to replicate the local data onto the remote side copying only that which is necessary. This is not necessarily the fastest, but it is the most efficient: rsync was designed for efficiency over slow links, not speed over high speed links. Configure rsync to use ssh encryption automatically in the Makefile, then use rsync as the way to copy the files over:

RSYNC_RSH=/usr/bin/ssh -i /path/to/mykey

rsync -av $(LOCAL_FILES) remoteserver:$(PWD)

To automate this properly, an ssh key will have to be created using keygen and transfered to the other host. The private key (/path/to/mykey in this example) is used by ssh in the background during rsync processing; with the key in place, no interactive login is necessary.

For best purposes, create an “all” tag (at the top of the file) that explains the usable tags, and create a “copy” tag that does the relevant rsync.

I recommend copying only relevant files, not the entire directory: this way, some files can be retained only on one node – this is good for log files and for temporary files.

For example:

LOCAL_FILES=*.pkg *.ctl *.m4
RSYNC_RSH=/usr/bin/ssh -i /path/to/mykey

all:
    echo "To copy files, use the copy tag..."

copy:
    rsync -av $(LOCAL_FILES) remserver:$(PWD)

Make sure to verify the code before you use it in normal operation. Use the rsync option -n to perform a dry run which affects nothing. Also make sure that you don’t update files on different hosts; things might get interesting (and unfortunate…)

After performing the update, the Makefile can trigger a reconfiguration or a reload of the daemons to put the configuration in place.