Ubuntu Backup Server 20.04 LTS

[LHammonds]

------------- WORK-IN-PROGRESS -------------

Greetings and salutations,

I hope this thread will be helpful to those who follow in my foot steps as well as getting any advice based on what I have done / documented.

To discuss this thread, please participate here: Ubuntu Forums

High-level overview

This server will be a repository (preferably offsite) for all files that need to be backed up from other servers. It will be configured to pull the local backup files on each remote server to its own repository. This design will ensure that the remote servers do not have access to the backup location which will enhance security of the backups even if the servers getting backed up are compromised.

NFS will be used to mount the remote servers and copy or move the designated files. Other mount options are available such as Samba if you need to connect to Windows servers and such.

There will be two main sets of local backups that will be collected. The first are partition backups which will remain on the original server at /bak/partitions (from the back-parts.sh job) and also have a copy on the remote backup server. The second will be data files that are placed by a local backup job into the /bak/remote folder which will be moved from the server.

Tools utilized in this process

• Ubuntu Server 20.04 LTS, 64-bit
• Portable PuTTY 0.73
• VMware vSphere 6.0.0
• VirtualBox 6.0.22

Helpful links

The list below are sources of information that was helpful in the creation of this document.

• Ubuntu Documentation

Assumptions

This documentation will need to make use of some very-specific information that will most-likely be different for each person / location. And as such, this information will be noted in this section. They will be highlighted in red throughout the document as a reminder that you should plug-in your own value rather than actually using these "place-holder" values.

Under no circumstance should you use the actual values listed below. They are place-holders for the real thing. This is just a checklist template you need to have answered before you start the install process.

Wherever you see RED in this document, you need to substitute it for you will use in your environment.

• Internet domain: srv-bak.mydomain.com
• Ubuntu Server name: srv-bak
• Ubuntu Server IP address: 192.168.107.50
• Ubuntu Admin ID: administrator
• Ubuntu Admin Password: myadminpass
• Email Server Name (remote): srv-mail
• Email Server IP (remote): 192.168.107.25

Backup Server - Setup an Ubuntu server for use as the backup server. This tutorial assumes the server was configured according to this tutorial: How to install Ubuntu server

It is also assumed the reader knows how to use the VI editor. If not, you will need to beef up your skill set or use a different editor in place of it.

[LHammonds]

Increase bak volume

Determine how much space you will need to hold the backup files and add the space to the /bak volume. In my case, I will add a 200 GB drive and allocate 100% of that space to the bak volume but only expand the file system by 80% of that to ensure there is room to expand in the future if space starts to get tight.

If you followed my instructions for setting up the server, there is a section that describes how to add more hard drives and allocated the space to the LVM, volume and how to expand the file system.

Once the extra space has been added to the LVM, you can use commands similar to these to increase the bak volume and file system:

Code: Select all

sudo lvextend -L200G /dev/LVG/bak
sudo resize2fs /dev/LVG/bak 160G

NFS Connectivity

Each remote server to be backed up needs to have NFS configured to share the /bak volume to just the backup server and ensure the firewall rules allow NFS connection for the backup server. The backup server only needs the ability to connect to the NFS shares.

NFS on Backup Server

On just the backup server, install the NFS client software:

Code: Select all

sudo apt install nfs-common

For every server that will be connected to, make sure it can resolve the name to an IP via DNS...or by editing the /etc/hosts file.

NFS on Source Servers

On each remote server to be backed up, install the NFS server software:

Code: Select all

sudo apt install nfs-kernel-server

Setup the export configuration file:

Code: Select all

sudo touch /etc/exports
sudo chown root:root /etc/exports
sudo chmod 644 /etc/exports
sudo vi /etc/exports

Add this line which will allow only the backup server to access the share (be sure to use your backup server IP instead of the example IP):

Code: Select all

/bak        192.168.107.50(rw,sync,no_root_squash,no_subtree_check)

Restart services for changes to take effect:

Code: Select all

sudo systemctl restart nfs-kernel-server

Modify your firewall script to open port 2049 TCP and UDP. Example:

Code: Select all

sudo vi /var/scripts/prod/en-firewall.sh

Code: Select all

ufw allow proto tcp from 192.168.107.50 to any port 2049 comment 'NFS Share'
ufw allow proto udp from 192.168.107.50 to any port 2049 comment 'NFS Share'

Run the script for the firewall changes to take effect:

Code: Select all

sudo /var/scripts/prod/en-firewall.sh

Check the status of the firewall rules and ensure the 2049 rules are in place:

Code: Select all

sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
22                         ALLOW       192.168.107.0/24           # SSH via LAN
2049/tcp                   ALLOW       192.168.107.50             # NFS Share
2049/udp                   ALLOW       192.168.107.50             # NFS Share

NFS Test Mount

On the backup server, create a mount point and mount a server to be backed up. In this example, it is a server called "srv-ftp" which has an entry in /etc/hosts

Code: Select all

sudo mkdir -p /mnt/srv-ftp
mount srv-ftp:/bak /mnt/srv-ftp

Look in the mounted folder and make sure you are seeing the contents of the /bak folder on the remote server and then disconnect once verified:

Code: Select all

ls -l /mnt/srv-ftp

Code: Select all

umount /mnt/srv-ftp

[LHammonds]

Transfer Script - Partition Copy

This script will run on a regular basis and copies the backed up partition files on each remote server. Make sure it is timed to run after the local back-parts.sh script is finished and not more often than that job runs.

Code: Select all

sudo touch /var/scripts/prod/transfer-partitions.sh
sudo chown root:root /var/scripts/prod/transfer-partitions.sh
sudo chmod 700 /var/scripts/prod/transfer-partitions.sh
sudo vi /var/scripts/prod/transfer-partitions.sh

/var/scripts/prod/transfer-partitions.sh (GitHub Download)

Code: Select all

#!/bin/bash
#############################################################
## Name          : transfer-partitions.sh
## Version       : 1.0
## Date          : 2020-06-01
## Author        : LHammonds
## Purpose       : Move backed up files from remote servers.
## Compatibility : Verified on Ubuntu Server 20.04 LTS
## Requirements  : None
## Run Frequency : Run as needed.
## Parameters    :
##    1 = (Required) Server name (needs to be resolvable to IP address)
## Exit Codes    :
##    0 = Success
##    1 = ERROR: Missing / Incorrect server name
##    2 = ERROR: Must run as root
##    4 = ERROR: Cannot resolve IP
##    8 = ERROR: LockFile detected
##   16 = ERROR: Mount failure
##   32 = ERROR: rsync failure
###################### CHANGE LOG ###########################
## DATE       VER WHO WHAT WAS CHANGED
## ---------- --- --- ---------------------------------------
## 2020-06-01 1.0 LTH Created script.
#############################################################

source /var/scripts/common/standard.conf

LogFile="${LogDir}/${Company}-transfer-partitions.log"
LockFile="${TempDir}/${Company}-transfer-partitions.lock"
SourceDir="/mnt/${1}"
TargetDir="/bak/${1}"
ErrorFlag=0
ReturnCode=0

#######################################
##            FUNCTIONS              ##
#######################################

function f_cleanup()
{
  if [ -f ${LockFile} ];then
    ## Remove lock file so other check space jobs can run.
    rm ${LockFile} 1>/dev/null 2>&1
  fi
  echo "`date +%Y-%m-%d_%H:%M:%S` [INFO] Exit code for ${ServerName}: ${ErrorFlag}" | tee -a ${LogFile}
  exit ${ErrorFlag}
}

function f_showhelp()
{
  echo -e "\n${LGREEN}Usage${COLORRESET}  : ${ScriptName} ${LYELLOW}{ServerName}${COLORRESET}"
  echo -e "${LGREEN}Example${COLORRESET}: ${ScriptName} ${LRED}srv-mariadb${COLORRESET}\n"
}

#######################################
##           MAIN PROGRAM            ##
#######################################

## Check existance of required command-line parameter(s).
case "$1" in
  "")
    f_showhelp
    ErrorFlag=2
    f_cleanup
    ;;
  --help|-h|-?)
    f_showhelp
    ErrorFlag=1
    f_cleanup
    ;;
  *)
    ServerName=$1
    ;;
esac

## Requirement Check: Script must run as root user.
if [ "$(id -u)" != "0" ]; then
  ## FATAL ERROR DETECTED: Document problem and terminate script.
  echo "[ERROR] Root user required to run this script."
  echo ""
  ErrorFlag=2
  f_cleanup
fi

## Check validity of server name.
## 0 = No associated IP address
## 1 = Found IP address
ReturnCode=`dig +short ${ServerName} | wc -l`
if [ ${ReturnCode} -eq 0 ]; then
  ## ERROR: ServerName not valid.
  echo "[ERROR] ${ServerName} cannot be resolved to an IP address." | tee -a ${LogFile}
  ErrorFlag=4
  f_cleanup
fi

if [ -f ${LockFile} ]; then
  # Lock file detected.  Abort script.
  echo "Script aborted"
  echo "This script tried to run but detected the lock file: ${LockFile}"
  echo "Please check to make sure the file does not remain when check space is not actually running."
  f_sendmail "ERROR: Transfer partitions script aborted" "This script tried to run but detected the lock file: ${LockFile}\n\nPlease check to make sure the file does not remain when check space is not actually running.\n\nIf you find that the script is not running/hung, you can remove it by typing 'rm ${LockFile}'"
  ErrorFlag=8
  f_cleanup
else
  echo "`date +%Y-%m-%d_%H:%M:%S` ${ScriptName}" > ${LockFile}
fi

## Check if the mount point exists.
if [ ! -d ${SourceDir} ]; then
  ## Create mount point and offline indicator file.
  ## NOTE: If you can see offline.txt, then its not mounted.
  mkdir -p ${SourceDir}
  echo "Offline test file" > ${SourceDir}/offline.txt
  chown root:root ${SourceDir}/offline.txt
  chmod 444 ${SourceDir}/offline.txt
fi
if [ ! -d ${TargetDir} ]; then
  ## Create the target folder destination.
  mkdir -p ${TargetDir}
  chown root:root ${TargetDir}
  chmod 700 ${TargetDir}
fi
echo "`date +%Y-%m-%d_%H:%M:%S` [INFO] Started - ${ServerName}" | tee -a ${LogFile}
## Connect to the server to pull files from.
mount ${ServerName}:/bak ${SourceDir}
## Make sure mount command was successful.
if [ -f ${SourceDir}/offline.txt ]; then
  ## Could not connect to remote server.
  echo "[ERROR] Not mounted: ${SourceDir}" | tee -a ${LogFile}
  ErrorFlag=16
  f_cleanup
else
  echo "`date +%Y-%m-%d_%H:%M:%S` [INFO] rsync ${SourceDir}/partitions ${TargetDir}" | tee -a ${LogFile}
  rsync -apogHK --out-format="%n" --delete --exclude=*.pid ${SourceDir}/partitions ${TargetDir} >> ${LogFile}
  ReturnCode=$?
  if [ ${ReturnCode} -ne 0 ]; then
    ## Fatal error detected.
    echo "`date +%Y-%m-%d_%H:%M:%S` [SEVERE] rsync failed ${SourceDir}/partitions to ${TargetDir}. Return Code = ${ReturnValue}" | tee -a ${LogFile}
    ErrorFlag=32
    umount ${SourceDir}
    f_cleanup
  fi
fi
echo "`date +%Y-%m-%d_%H:%M:%S` [INFO] Completed - ${ServerName}" | tee -a ${LogFile}
## Disconnect from the server.
umount ${SourceDir}
f_cleanup

Example log output: /var/log/abc-transfer-partitions.log

Code: Select all

2020-06-01_11:14:25 [INFO] Started - srv-database
2020-06-01_11:14:25 [INFO] rsync /mnt/srv-database/partitions /bak/srv-database
partitions/
partitions/srv-database-boot.fsa
partitions/srv-database-boot.md5
partitions/srv-database-boot.txt
partitions/srv-database-home.fsa
partitions/srv-database-home.md5
partitions/srv-database-home.txt
partitions/srv-database-root.fsa
partitions/srv-database-root.md5
partitions/srv-database-root.txt
partitions/srv-database-tmp.fsa
partitions/srv-database-tmp.md5
partitions/srv-database-tmp.txt
partitions/srv-database-var.fsa
partitions/srv-database-var.md5
partitions/srv-database-var.txt
2020-06-01_11:14:35 [INFO] Completed - srv-database
2020-06-01_11:14:35 [INFO] Exit code for srv-database: 0

Example crontab schedule:

Code: Select all

0 4 1 * * /var/scripts/prod/transfer-partitions.sh srv-database > /dev/null 2>&1
5 4 1 * * /var/scripts/prod/transfer-partitions.sh srv-wordpress > /dev/null 2>&1
10 4 1 * * /var/scripts/prod/transfer-partitions.sh srv-sftp > /dev/null 2>&1
15 4 1 * * /var/scripts/prod/transfer-partitions.sh srv-nagios > /dev/null 2>&1
20 4 1 * * /var/scripts/prod/transfer-partitions.sh srv-nextcloud > /dev/null 2>&1
25 4 1 * * /var/scripts/prod/transfer-partitions.sh srv-xibo > /dev/null 2>&1
30 4 1 * * /var/scripts/prod/transfer-partitions.sh srv-php > /dev/null 2>&1

[LHammonds]

Transfer Script - Data Move

This script will run on a regular basis and move the data files on each remote server that placed files in the /bak/remote folder.

Code: Select all

sudo touch /var/scripts/prod/transfer-data.sh
sudo chown root:root /var/scripts/prod/transfer-data.sh
sudo chmod 700 /var/scripts/prod/transfer-data.sh
sudo vi /var/scripts/prod/transfer-data.sh

/var/scripts/prod/transfer-data.sh (GitHub Download)

Code: Select all

#!/bin/bash
#############################################################
## Name          : transfer-data.sh
## Version       : 1.0
## Date          : 2020-06-01
## Author        : LHammonds
## Purpose       : Move backed up files from remote servers.
## Compatibility : Verified on Ubuntu Server 20.04 LTS
## Requirements  : None
## Run Frequency : Run as needed.
## Parameters    :
##    1 = (Required) Server name (needs to be resolvable to IP address)
## Exit Codes    :
##    0 = Success
##    1 = ERROR: Missing / Incorrect server name
##    2 = ERROR: Must run as root
##    4 = ERROR: Cannot resolve IP
##    8 = ERROR: LockFile detected
##   16 = ERROR: Mount failure
##   32 = ERROR: rsync failure
###################### CHANGE LOG ###########################
## DATE       VER WHO WHAT WAS CHANGED
## ---------- --- --- ---------------------------------------
## 2020-06-01 1.0 LTH Created script.
#############################################################

## Import common variables and functions. ##
source /var/scripts/common/standard.conf

LogFile="${LogDir}/${Company}-transfer-data.log"
LockFile="${TempDir}/${Company}-transfer-data.lock"
SourceDir="/mnt/${1}"
TargetDir="/bak/${1}"
ErrorFlag=0
ReturnCode=0
DaysToKeep=15

#######################################
##            FUNCTIONS              ##
#######################################

function f_cleanup()
{
  if [ -f ${LockFile} ];then
    ## Remove lock file so other check space jobs can run.
    rm ${LockFile} 1>/dev/null 2>&1
  fi
  echo "`date +%Y-%m-%d_%H:%M:%S` [INFO] Exit code for ${ServerName}: ${ErrorFlag}" | tee -a ${LogFile}
  exit ${ErrorFlag}
}

function f_showhelp()
{
  echo -e "\n${LGREEN}Usage${COLORRESET}  : ${ScriptName} ${LYELLOW}{ServerName}${COLORRESET}"
  echo -e "${LGREEN}Example${COLORRESET}: ${ScriptName} ${LRED}srv-mariadb${COLORRESET}\n"
}

#######################################
##           MAIN PROGRAM            ##
#######################################

## Check existance of required command-line parameter(s).
case "$1" in
  "")
    f_showhelp
    ErrorFlag=2
    f_cleanup
    ;;
  --help|-h|-?)
    f_showhelp
    ErrorFlag=1
    f_cleanup
    ;;
  *)
    ServerName=$1
    ;;
esac

## Requirement Check: Script must run as root user.
if [ "$(id -u)" != "0" ]; then
  ## FATAL ERROR DETECTED: Document problem and terminate script.
  echo "[ERROR] Root user required to run this script."
  echo ""
  ErrorFlag=2
  f_cleanup
fi

## Check validity of server name.
## 0 = No associated IP address
## 1 = Found IP address
ReturnCode=`dig +short ${ServerName} | wc -l`
if [ ${ReturnCode} -eq 0 ]; then
  ## ERROR: ServerName not valid.
  echo "[ERROR] ${ServerName} cannot be resolved to an IP address." | tee -a ${LogFile}
  ErrorFlag=4
  f_cleanup
fi

if [ -f ${LockFile} ]; then
  # Lock file detected.  Abort script.
  echo "Script aborted"
  echo "This script tried to run but detected the lock file: ${LockFile}"
  echo "Please check to make sure the file does not remain when check space is not actually running."
  f_sendmail "ERROR: Transfer data script aborted" "This script tried to run but detected the lock file: ${LockFile}\n\nPlease check to make sure the file does not remain when check space is not actually running.\n\nIf you find that the script is not running/hung, you can remove it by typing 'rm ${LockFile}'"
  ErrorFlag=8
  f_cleanup
else
  echo "`date +%Y-%m-%d_%H:%M:%S` ${ScriptName}" > ${LockFile}
fi

## Check if the mount point exists.
if [ ! -d ${SourceDir} ]; then
  ## Create mount point and offline indicator file.
  ## NOTE: If you can see offline.txt, then its not mounted.
  mkdir -p ${SourceDir}
  echo "Offline test file" > ${SourceDir}/offline.txt
  chown root:root ${SourceDir}/offline.txt
  chmod 444 ${SourceDir}/offline.txt
fi
if [ ! -d ${TargetDir} ]; then
  ## Create the target folder destination.
  mkdir -p ${TargetDir}
  chown root:root ${TargetDir}
  chmod 700 ${TargetDir}
fi
echo "`date +%Y-%m-%d_%H:%M:%S` [INFO] Started - ${ServerName}" | tee -a ${LogFile}
## Connect to the server to pull files from.
mount ${ServerName}:/bak ${SourceDir}
## Make sure mount command was successful.
if [ -f ${SourceDir}/offline.txt ]; then
  ## Could not connect to remote server.
  echo "[ERROR] Not mounted: ${SourceDir}" | tee -a ${LogFile}
  ErrorFlag=16
  f_cleanup
fi

## Purge any archives older than x days
find ${TargetDir}/remote -maxdepth 1 -type d -mtime +${DaysToKeep} -exec -rf {} \;
echo "`date +%Y-%m-%d_%H:%M:%S` [INFO] rsync ${SourceDir}/remote ${TargetDir}" | tee -a ${LogFile}
rsync -apogHK --out-format="%n" --exclude=*.pid ${SourceDir}/remote ${TargetDir} >> ${LogFile}
ReturnCode=$?
if [ ${ReturnCode} -ne 0 ]; then
  ## Fatal error detected.
  echo "`date +%Y-%m-%d_%H:%M:%S` [SEVERE] rsync failed ${SourceDir} to ${TargetDir}. Return Code = ${ReturnValue}" | tee -a ${LogFile}
  ErrorFlag=32
  umount ${SourceDir}
  f_cleanup
fi
## Remove the archive(s) that were just transferred.
echo "`date +%Y-%m-%d_%H:%M:%S` [INFO] Removing ${SourceDir}/remote/*" | tee -a ${LogFile}
rm -rf ${SourceDir}/remote/*
echo "`date +%Y-%m-%d_%H:%M:%S` [INFO] Completed - ${ServerName}" | tee -a ${LogFile}

## Disconnect from the server.
umount ${SourceDir}
f_cleanup

Example log output: /var/log/abc-transfer-data.log

Code: Select all

2020-06-01_10:23:13 [INFO] Started - srv-database
2020-06-01_10:23:13 [INFO] rsync /mnt/srv-database/remote /bak/srv-database
remote/
remote/2020-05-30-databases.tar.gz
2020-06-01_10:23:13 [INFO] Removing /mnt/srv-database/remote/*
2020-06-01_10:23:13 [INFO] Completed - srv-database
2020-06-01_10:23:13 [INFO] Exit code for srv-database: 0