How to Install and Configure an Ubuntu Server 20.04 LTS

User avatar
LHammonds
Site Admin
Site Admin
Posts: 877
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Partition-Level Restores

Post: # 834Post LHammonds »

Restore Partitions Using SystemRescueCD and FSArchiver

Partitions cannot be mounted when restoring to them. If services can be stopped that use files on a specific partition, it can be unmounted and restored. However, the root partition can never be restored while the server is online so these instructions will cover the common denominator which requires taking the server offline.

The server needs to be booted up with a CD but not just any CD will do because it needs to have FSArchiver on it. For this document, the ISO image from www.sysresccd.org will be used.

Once downloaded, the ISO can be burned to a CD-ROM disc or uploaded to your ISO repository such as a LUN. VMware and VirtualBox can attach an ISO image and mount it in the CD-ROM device to allow the virtual machine to boot the ISO image.

For this example, files on the root and home partitions will be restored using this summarized procedure:
  1. Boot the system using the RescueCD.
  2. Mount the bak partition which contains the backup archives.
  3. Verify that the backup archives are in good condition.
  4. Restore the partitions.
  5. Mount the root and home partitions.
  6. Verify the restore worked.
  7. Unmount the partitions.
  8. Reboot the system.
Be sure the partitions have been backed up and the files are sitting in the /bak volume.

Code: Select all

ls -l /bak
total 1035280 drwx------ 2 root root 16384 Apr 30 13:31 lost+found -rw------- 1 root root 156167681 May 3 15:33 srv-ubuntu-boot.fsa -rw------- 1 root root 59 May 3 15:33 srv-ubuntu-boot.md5 -rw------- 1 root root 747 May 3 15:33 srv-ubuntu-boot.txt -rw------- 1 root root 11113 May 3 15:40 srv-ubuntu-home.fsa -rw------- 1 root root 59 May 3 15:40 srv-ubuntu-home.md5 -rw------- 1 root root 749 May 3 15:40 srv-ubuntu-home.txt -rw------- 1 root root 837941885 May 3 15:38 srv-ubuntu-root.fsa -rw------- 1 root root 59 May 3 15:38 srv-ubuntu-root.md5 -rw------- 1 root root 750 May 3 15:38 srv-ubuntu-root.txt -rw------- 1 root root 7142 May 3 15:40 srv-ubuntu-tmp.fsa -rw------- 1 root root 58 May 3 15:40 srv-ubuntu-tmp.md5 -rw------- 1 root root 745 May 3 15:40 srv-ubuntu-tmp.txt -rw------- 1 root root 65930756 May 3 15:40 srv-ubuntu-var.fsa -rw------- 1 root root 58 May 3 15:40 srv-ubuntu-var.md5 -rw------- 1 root root 748 May 3 15:40 srv-ubuntu-var.txt
As a test of the restore, let's delete the two text files created in the previous section:

Code: Select all

rm /important.txt
rm /home/administrator/important.txt
These files should have been included in the backup image. When the restore is complete, these files should return.

Insert the SystemRescueCD CDROM (or mount the ISO image) and boot the server with it.

Here is the 1st screen:
Image

Option #1 was chosen (SystemRescueCd: default boot options)

The next screen takes you to the command prompt:

Image

Show what partitions the system has by using fsarchiver.

root@sysresccd /root % fsarchiver probe simple
[======DISK======] [=============NAME==============] [====SIZE====] [MAJ] [MIN] [sda ] [VBOX HARDDISK ] [ 25.00 GB] [ 8] [ 0] [sr0 ] [CD-ROM ] [ 675.00 MB] [ 11] [ 0] [=====DEVICE=====] [==FILESYS==] [======LABEL======] [====SIZE====] [MAJ] [MIN] [loop0 ] [squashfs ] [<unknown> ] [ 620.12 MB] [ 7] [ 0] [sda1 ] [ext4 ] [boot ] [ 953.00 MB] [ 8] [ 1] [sda5 ] [LVM2_member] [<unknown> ] [ 24.87 GB] [ 8] [ 5] [dm-0 ] [ext4 ] [root ] [ 6.00 GB] [254] [ 0] [dm-1 ] [ext4 ] [var ] [ 3.00 GB] [254] [ 1] [dm-2 ] [ext4 ] [tmp ] [ 1.00 GB] [254] [ 2] [dm-3 ] [ext4 ] [bak ] [ 4.00 GB] [254] [ 3] [dm-4 ] [ext4 ] [home ] [ 1.00 GB] [254] [ 4]
Mount the bak partition which contains the backup archives.

root@sysresccd /root % mkdir /bak
root@sysresccd /root % mount --read-only /dev/dm-3 /bak
root@sysresccd /root % ls -l /bak
total 1035280 drwx------ 2 root root 16384 Apr 30 13:31 lost+found -rw------- 1 root root 156167681 May 3 15:33 srv-ubuntu-boot.fsa -rw------- 1 root root 59 May 3 15:33 srv-ubuntu-boot.md5 -rw------- 1 root root 747 May 3 15:33 srv-ubuntu-boot.txt -rw------- 1 root root 11113 May 3 15:40 srv-ubuntu-home.fsa -rw------- 1 root root 59 May 3 15:40 srv-ubuntu-home.md5 -rw------- 1 root root 749 May 3 15:40 srv-ubuntu-home.txt -rw------- 1 root root 837941885 May 3 15:38 srv-ubuntu-root.fsa -rw------- 1 root root 59 May 3 15:38 srv-ubuntu-root.md5 -rw------- 1 root root 750 May 3 15:38 srv-ubuntu-root.txt -rw------- 1 root root 7142 May 3 15:40 srv-ubuntu-tmp.fsa -rw------- 1 root root 58 May 3 15:40 srv-ubuntu-tmp.md5 -rw------- 1 root root 745 May 3 15:40 srv-ubuntu-tmp.txt -rw------- 1 root root 65930756 May 3 15:40 srv-ubuntu-var.fsa -rw------- 1 root root 58 May 3 15:40 srv-ubuntu-var.md5 -rw------- 1 root root 748 May 3 15:40 srv-ubuntu-var.txt
Verify that the backup archives are in good condition:

root@sysresccd /root % md5sum --check /bak/srv-ubuntu-root.md5
/bak/srv-ubuntu-root.fsa: OK
root@sysresccd /root % md5sum --check /bak/srv-ubuntu-home.md5
/bak/srv-ubuntu-home.fsa: OK
Restore the root partition:

root@sysresccd /root % fsarchiver restfs --cryptpass="abc123" /bak/srv-ubuntu-root.fsa id=0,dest=/dev/dm-0
Statistics for filesystem 0 * files successfully processed:....regfiles=96032, directories=19993, symlinks=7266, hardlinks=9, specials=8 * files with errors:...............regfiles=0, directories=0, symlinks=0, hardlinks=0, specials=0
Restore the home partition:

root@sysresccd /root % fsarchiver restfs --cryptpass="abc123" /bak/srv-ubuntu-home.fsa id=0,dest=/dev/dm-4
Statistics for filesystem 0 * files successfully processed:....regfiles=11, directories=5, symlinks=0, hardlinks=0, specials=0 * files with errors:...............regfiles=0, directories=0, symlinks=0, hardlinks=0, specials=0
Mount the root and home partitions:

root@sysresccd /root % mkdir /mnt/root
root@sysresccd /root % mkdir /mnt/home
root@sysresccd /root % mount --read-only /dev/dm-0 /mnt/root
root@sysresccd /root % mount --read-only /dev/dm-4 /mnt/home

Verify the restore worked:

root@sysresccd /root % ls -l /mnt/root/import*
-rw-r--r-- 1 root root 0 May 3 17:00 important.txt
root@sysresccd /root % ls -l /mnt/home/administrator/import*
-rw-r--r-- 1 root root 0 May 3 17:00 important.txt
Unmount the partitions:

root@sysresccd /root % umount /mnt/root
root@sysresccd /root % umount /mnt/home
root@sysresccd /root % umount /bak

Eject the CDROM/ISO and reboot the server.

If it worked right, the server will boot up (we restored the root file system) and the test files should have been restored.

NOTE: If you noticed any ext file systems that had <unknown> labels, you can update them using the tune2fs command which makes identification easier when restoring.

Example 1:

Code: Select all

sudo tune2fs /dev/sda1 -L boot
Example 2:

Code: Select all

sudo tune2fs /dev/dm-0 root

User avatar
LHammonds
Site Admin
Site Admin
Posts: 877
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Migrating Users from one server to another

Post: # 835Post LHammonds »

Migrating users/groups from one server to another

WARNING: These migration commands are thoroughly tested by me...just jotting down these research notes for future testing.

Commands to run on old server

Copy accounts to passwd.mig using awk to filter out system accounts (i.e. only copy user accounts)

Code: Select all

sudo awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534)' /etc/passwd > /tmp/passwd.mig
Copy groups to group.mig

Code: Select all

sudo awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534)' /etc/group > /tmp/group.mig
Copy shadow to shadow.mig

Code: Select all

sudo awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534) {print $1}' /etc/passwd | tee - |egrep -f - /etc/shadow > /tmp/shadow.mig
Copy gshadow to gshadow.mig (rarely used):

Code: Select all

sudo cp /etc/gshadow /tmp/gshadow.mig
Backup the home directories.

Code: Select all

sudo tar -zcvpf /tmp/home.tar.gz /home
If you have Samba users...

Code: Select all

sudo cp /etc/samba/smb.conf /tmp/smb.mig
sudo cp /etc/samba/smbusers /tmp/smbusers.mig
sudo cp /etc/samba/smbpasswd /tmp/smbpasswd.mig
Transfer the migration files to new server.

Code: Select all

sudo scp /tmp/*.mig administrator@newserver:/tmp/.
sudo scp /tmp/home.tar.gz administrator@newserver:/tmp/.
Commands to run on new server

Backup current files (and their backups).

Code: Select all

sudo mkdir /root/save
sudo cp /etc/passwd /etc/passwd- /etc/shadow /etc/shadow- /etc/group /etc/group- /etc/gshadow /etc/gshadow- /root/save
If you have Samba users, backup your current files.

Code: Select all

sudo cp /etc/samba/smb.conf /etc/samba/smbusers /etc/samba/smbpasswd /root/save
Merge/Overwrite files.

Code: Select all

sudo cat /tmp/passwd.mig >> /etc/passwd
sudo cat /tmp/group.mig >> /etc/group
sudo cat /tmp/shadow.mig >> /etc/shadow
sudo cp /tmp/gshadow.mig /etc/gshadow
Ensure file ownership and permissions are correct.

Code: Select all

sudo chown root:shadow /etc/shadow /etc/gshadow
sudo chmod 640 /etc/shadow /etc/gshadow
sudo chown root:root /etc/passwd /etc/group
sudo chmod 644 /etc/passwd /etc/group
If you have Samba users...

Code: Select all

sudo cp /tmp/smb.mig /etc/samba/smb.conf
sudo cp /tmp/smbusers.mig /etc/samba/smbusers
sudo cp /tmp/smbpasswd.mig /etc/samba/smbpasswd
Extract home directories.

Code: Select all

cd /
sudo tar -zxvf /tmp/home.tar.gz
Reboot system for new user accounts to take effect.

Code: Select all

sudo reboot

User avatar
LHammonds
Site Admin
Site Admin
Posts: 877
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Server Replacement

Post: # 836Post LHammonds »

Replacing a server with a new one

If you want to keep SSH keys and the fingerprint from changing when swapping out an old server for a new one (e.g. old FTPS server to new FTPS which interfaces with many 3rd-party vendors), these are the steps needed to keep the system looking the same.

The old server name and the new server name must match. Example: srv-ftps

Make sure the sshd configuration files on both servers are pointing to the same files (filenames and paths)

Code: Select all

sudo vi /etc/ssh/sshd_config
Look for section like this:

Code: Select all

# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
If there are differences, figure out if they matter and if so, make sure the transferred files on the new server are being referenced correctly in the config file on the new server.

Commands to run on old server

Transfer files to new server

Code: Select all

sudo scp /etc/ssh/ssh_host* administrator@newserver:/tmp/.
Commands to run on new server

Backup existing files

Code: Select all

sudo mkdir /root/ssh-backup
sudo cp /etc/ssh/ssh_host* /root/ssh-backup/.
Set correct file ownership and permissions

Code: Select all

sudo chown root:root /tmp/ssh_host*
sudo chmod 600 /tmp/ssh_host*
sudo chmod 644 /tmp/ssh_host*.pub
Move and overwrite files to correct location

Code: Select all

sudo mv /tmp/ssh_host* /etc/ssh/.
Restart SSH daemon

Code: Select all

sudo systemctl restart ssh
You will also want to copy over individual user's public keys in their ~/.ssh/authorized_keys but if you transferred the contents of the /home folder, this was already done.

User avatar
LHammonds
Site Admin
Site Admin
Posts: 877
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Trouble-Shooting

Post: # 837Post LHammonds »

Boot Issues

Code: Select all

systemd-analyze plot > /srv/samba/share/boot.xml
Open boot.xml in a web browser to examine the boot process to see if anything is taking longer than it should.

Code: Select all

systemd-analyze blame
This shows a list of all running processes ordered by the time it took to initialize. Be careful not to judge a process too harshly at 1st sight...it may have been waiting on another process to initialize 1st.

Code: Select all

dmesg > /tmp/boot.txt
Open boot.txt and observe the order of events from the moment it starts its boot all the way to the end. The events are ordered by time and it shows how much time has passed since the machine was turned on for that particular event to occur.

Disk Space Issues

To quickly find out status on partition sizes, use this command:

Code: Select all

df -h
To help find out what is eating up your space in a particular folder, you can use this command:

Code: Select all

du -sh /path/to/folder
To make faster use of du, use the ncdu utility I recommended to install in an earlier section. If you start at the root like the example below, you can use your arrow keys to select the problematic folders and keep drilling down until you find the culprit you are looking for. It automatically sorts the results so you are typically already on the biggest folder and just need to press ENTER to drill down to the next folder.

Code: Select all

ncdu /

User avatar
LHammonds
Site Admin
Site Admin
Posts: 877
Joined: Fri Jul 31, 2009 6:27 pm
Are you a filthy spam bot?: No
Location: Behind You
Contact:

Future Research Items

Post: # 838Post LHammonds »

Security Topics to Research and Incorporate
Gamin File Change Monitoring - Gamin web site
PSAD intrusion detection - TheFanClub walkthru (Note, current version is 2.4.4 released on Feb 2017, Ubuntu repository version is 2.2.3 released in March, 2014)
Tiger intrusion detection - homepage
Tripwire intrusion detection - DigitalOcean VPN, Unixmen
LogWatch - DigitalOcean walkthru, Unixmen, linode
Passing Passwords
Let's Encrypt (FREE SSL) and Certbot
etckeeper for config revision control

Post Reply