Jump To: Support > KB > Backup > Runrsync
NetManager backup server overview
Jump to: Backup structure | Backup log file | Excluding files and folders
Multi-generation efficient file copies
NetManager Backup uses rsync to copy files from the source. rsync copies only files that have changed and, when possible, only the parts of the files that differ. Files are pulled by the NetManager backup at a scheduled time; the server being backed up does not push the data to the backup server. This means that servers to be backed up need to run a backup service. Communication between the backup server and the device being backed up can be compressed and certain files can be skipped. The backup server uses a username and password to identify itself to the device being backed up. The username and password should be the same throughout your network, but this is separate from all other users (i.e. this is not an Active Directory or NetManager user). If some of the servers to be backed up are outside of your administrative control, you can specify an alternative username and password for specific servers.
The runrsync command performs the backup and if run without any options will backup all configured shares on all servers. The syntax at the time of writing is:
$Revision: 1.59 $
Syntax: runrsync [-ehlLnvqsrRot] [-m <email address>] <service ...>
-d = output log to stdout once backup completed (useful for email
from cron)
-e = update and edit configuration file, then quit
-h = this help
-l = list shares to be backed up, then quit
-L = list all files, but don't copy
-m = Email log file to specified or pre-configured address.
This option requires the server to be able to email via an
SMTP server. You can specify this option more than once
to send to multiple email addresses
-n = do a dry-run, but don't actually copy or delete files
-o = backup to remote server over ssh
(uses /etc/netmanager/rsync.conf)
-p = remove previous folders over a given number of days old
-P = as -p but quit without running backup
-r = restart transfer on data timeout
-R = number of times to restart (default 20, 0 = for ever)
-s = back up to local machine
(uses /etc/netmanager/localbackup.conf)
-S = copy symbolic links as files (only for settings share)
-t = timeout setting in seconds
(default 45 (300 if -o), 0 = no timeout)
-v = lists all files (adding more increases diagnostics)
service names are in the format "share@server"
runrsync syncs to configured list of rsync shares
The runrsync can run in 3 distinct modes (backup from remote servers to this server, backup this server locally and backup this server to remote server) and the configuration file for each is different, but they share the same general syntax.
Backup from remote servers
The file /etc/netmanager/runrsync.conf is used to configure the operation when backing up remote servers (this is the default mode when neither of the -s and -o options are given). When runrsync is first run, a blank configuration file will be created ready for customisation. The file is well-commented (lines beginning with # are comments). An example file is below:
# Created by runrsync : 1.59 $ # # Username to use to connect to rsync server # This is username defined by the server just for rsync (i.e. not a local user) user="backupuser" # Password to use to connect to rsync server # This is set in the rsyncd.secrets file password="mypassword" # Username and password can be overridden for certain servers # Format is space-separated list of servername=user:password # Any server not listed will use default user and password above useroverride="" # Shares and servers to backup # Specify just the server name to backup all available rsync shares # or use share@server to backup particular ones. shares="netmanager fileserver" # Local directory to backup to dest="/data/backup" # How to handle previous versions of files. If unset (not empty), a relative # folder called 'previous' will used (for compatibility). Leave empty to # generate no previous versions. Paths starting with / are relative to dest # set above and have the server/share name added. # "/%DAY" or "backup" are the recommended values. # There are a number of substitutions that will be made: # Use %DAY for the day of the week # Use %MONTH for the month as two digits (e.g. 01) # Use %YEAR for the year as four digits (e.g. 2012) # Use %WEEK for the week of the year # Use %WEEK2 for alternate weeks (i.e. 1 or 2 - allows fortnightly backups) # Use %WEEK4 for a 4-week cycle (i.e. 1, 2, 3 or 4 - allows 28-day backups) # e.g. /%DAY-%WEEK2 will give 14 backups with names Mon-1, Tue-1 through to # Sun-2. # Use %DATE for the actual date (2010-06-13) # Substitutions can be combined (e.g. /previous/%YEAR/%DATE) previous="/%DATE" # Time to wait with no activity before aborting the backup in seconds. # Used to spot a failed connection, but should not be too short as otherwise # the scanning phase of the backup can cause a timeout. # Default = 45, but should be increased if very large files or number of files # on server being backed up # Set to 0 to disable timeout timeoutsecs=300 # Number of days to keep previous version of files for. Only makes sense if # 'previous' setting above has %DATE in it. Age will be calculated from last # successful backup rather than current date # Set to 0 or leave empty to not expire at all expire=90 # Delete files from the backup that no longer exist? (Default: yes) delete="y" # File extensions to skip (space separated) skip="" # Email address to email log file to email="" # Compress transfer? compress="n"
This configuration backs up all configured shares (configured in the backup agent, not standard Windows shares) on the servers called netmanager and fileserver.
Correct operation can checked by using the -l flag:
backup 1# runrsync -l home@netmanager settings@netmanager Users@fileserver Common@fileserver backup 2#
Here we can see that the 2 servers each have 2 shared resources being backed up (earlier version of runrsync will show this as home on netmanager, etc. This has been changed so that the lines show exactly what you can specify on the command line to backup just that share.
Backup structure
After the backup has been run a few times, the following structure will be seen:
The backup structure is split between the Live backup (which will reflect the state when the backup was last run, i.e. usually last night) and the Previous versions. For the Live backups, each server has its own folder named after that server. The Previous versions are all in dated folders underneath a folder called previous. To understand how the live and previous versions work, it is useful to know how the backup process works.
When the backup runs, all folders and files will be compared between the server being backed up and versions in the Live backup. If files have been changed or deleted, the old version will be moved to a folder named after the current date (the path within that dated folder will reflect the location the file exists at in the live version, so for example if the live file is fileserver/shared/Document.odt, the old version will be moved to previous/YYYY-MM-DD/fileserver/shared/Document.odt). The live version will then be updated from the current version of the file (or removed if the file has been deleted).
As files only get moved to the dated Previous folders when they change, files that do not change will not be in the Previous folders (and also therefore will take up no more space). The size of a dated Previous folder represents the size of the files changed within that day (assuming backups are run nightly).
Backup log file
Each dated previous folder contains a report of the status of the backup on that date called backuplog.txt. The lastrun.txt file at the top level is actually a link to the most recent backuplog.txt file. An example extract is:
Last backup run started: Tue Nov 30 08:41:58 GMT 2010 Backup : home on netmanager -------------------------------------------------- Started : Tue Nov 30 08:41:58 GMT 2010 Completed : Tue Nov 30 08:41:59 GMT 2010 Backup was **successful** Previous versions backed up to: /2010-11-30 Number of files: 9588 Number of files transferred: 0 Total file size: 1.35G bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 191.40K File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 3.40K Total bytes received: 202.14K sent 3.40K bytes received 202.14K bytes 137.02K bytes/sec total size is 1.35G speedup is 6563.23
Excluding files and paths
For a blanket block on certain file types, just add the file extensions to the skip line (e.g. skip="mp3 avi mpg").
For blocking paths and files use the following files:
/etc/netmanager/backup.exclude- global for all backup types/etc/netmanager/localbackup.exclude- additional exclusions with -s option (backup to local storage)/etc/netmanager/externalbackup.exclude- additional exclusions with -o option (backup to remote external system)
Each line contains a case-sensitive match against the root of the files in the share. Directory separator is /, not \. Wildcard can be specified with * (lines do not contain regular expressions).
So, for example, to block all .mp3 files you can use:*/*.mpgTo block all directories with names containing Old use:
*/*Old*/*
Here's an example /etc/netmanager/externalbackup.exclude to stop sent mail being backed up off-site to save space:
*/Sent */Sent Messages */SentMail */sent-mail
Lines to act on a particular server should be prefixed with [server], lines to act on a particular share on a particular server should be prefixed with [share@server]. All other lines will act on all servers. For example, the following will skip all mp3 files, but only skip .avi files on the server called resources:
*/*.mp3 [resources]*/*.avi
NFS
Configuration of NFS storage will be specific to the site.
SMB/CIFS
Configuration of SMB/CIFS storage will be specific to the site.